Edge control method and device for operating machinery, operating machinery and electronic equipment

Abstract

The present invention provides a method and device for controlling the edge of a working machine, a working machine and an electronic device, wherein the method for controlling the edge includes: obtaining image information of the side of the working machine and point cloud data of the side of the working machine; determining classification and recognition results based on the image information; determining a distance measurement algorithm based on the classification and recognition results; using the distance measurement algorithm to process the point cloud data, and determining the edge distance and running posture of the working machine, wherein the edge distance and the running posture are used for edge control. The method for controlling the edge of the present invention collects environmental images on both sides of the working machine and performs feature recognition, and selects a corresponding distance measurement algorithm based on the feature recognition results to avoid the influence of obstacle types on calculation errors, effectively improving the accuracy of the edge distance measurement results and the universality of the method; outputting the edge distance and running posture to jointly characterize the real-time position and posture of the working machine to achieve automatic edge control or assist manual driving.

Description

Technical Field

[0001] The present invention relates to the technical field of working machinery, and in particular to a method and device for controlling the edge of a working machinery, and a working machinery and electronic equipment. Background Art

[0002] Operating machinery is widely used in road construction and other operating scenarios. Especially in compaction and other operating scenarios that require edge-to-edge operations, there are often high requirements for edge-to-edge accuracy. With the development of automatic control technology, operating machinery has begun to realize automatic edge-to-edge control, which has higher requirements for edge-to-edge accuracy.

[0003] In the prior art, edge control is mainly performed by GPS positioning, manual observation with a cursor, or observation with a panoramic camera. The edge control performed by these technologies often has low operating accuracy and cannot meet the requirements of unmanned operation. Summary of the invention

[0004] The present invention provides a method and device for controlling the edge of an operating machine, an operating machine and an electronic device, which are used to at least partially solve or improve the defects in the prior art that the edge of the operating machine is not accurate and the operation is complicated for operators when performing edge operations, and realize high-precision and simplified edge operations.

[0005] The present invention provides a method for controlling the edge of a working machine, comprising:

[0006] Acquire image information of the side surface of the working machine and point cloud data of the side surface of the working machine;

[0007] Based on the image information, determining a classification and recognition result;

[0008] Determining a distance measurement algorithm based on the classification and recognition result;

[0009] The distance measurement algorithm is used to process the point cloud data to determine the edge distance and running posture of the working machine, and the edge distance and running posture are used for edge control.

[0010] According to a method for controlling the edge of a working machine provided by the present invention, the working machine comprises a first sensor and a second sensor arranged at intervals in a front-to-rear direction;

[0011] The method of using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine includes:

[0012] Determining a first edge distance of the operating machine based on the point cloud data collected by the first sensor;

[0013] Determining a second edge distance of the operating machine based on the point cloud data collected by the second sensor;

[0014] The walking posture is determined based on the first edge distance and the second edge distance.

[0015] According to a method for controlling the edge of a working machine provided by the present invention, before using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine, the method further includes:

[0016] Taking the target point of the working machine as the origin, setting a reference coordinate system;

[0017] The point cloud data is transformed into the reference coordinate system.

[0018] According to a method for controlling the edge of a working machine provided by the present invention, the reference coordinate system includes an X direction along the traveling direction, a Y direction along the horizontal direction, and a Z direction along the vertical direction;

[0019] When the classification and recognition result is a curb, the using of the distance measurement algorithm to process the point cloud data and determine the edge distance of the working machine includes:

[0020] Based on the height target range corresponding to the classification recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain first point cloud data;

[0021] Clustering the first point cloud data into one category, removing point clouds beyond a first target distance from a cluster center, and obtaining second point cloud data;

[0022] Determining the edge distance based on the second point cloud data;

[0023] Alternatively, when the classification and recognition result is a railing, using the distance measurement algorithm to process the point cloud data to determine the edge distance of the working machine includes:

[0024] Based on the height target range corresponding to the classification recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain third point cloud data;

[0025] Extracting point cloud data based on the Z direction from the third point cloud data, and fitting the extracted point cloud data to obtain a point cloud curve function;

[0026] Determining the edge distance based on the point cloud curve function;

[0027] Alternatively, when the classification and recognition result is other, the using of the distance measurement algorithm to process the point cloud data to determine the edge distance of the working machine includes:

[0028] Based on the distance target range and the value of the point cloud data in the Y direction, multiple categories of clustering are performed, and the point cloud whose cluster center is closest to the operating machinery is selected as the candidate point cloud data;

[0029] The edge distance is determined based on the candidate point cloud data.

[0030] According to a method for controlling the edge of a working machine provided by the present invention, the step of determining a classification and recognition result based on the image information includes:

[0031] The image information is input into an image recognition model, and the classification recognition result corresponding to the image information is output; wherein

[0032] The image recognition model is a neural network model, which is trained using sample image information as samples and predetermined classification recognition results corresponding to the sample image information as sample labels.

[0033] According to a method for controlling the edge of a working machine provided by the present invention, after using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine, the method further includes:

[0034] Based on the edge distance and the running posture, controlling the steering system of the working machine;

[0035] or,

[0036] Based on the edge distance and the running posture, driving assistance information is output.

[0037] The present invention also provides an edge control device for a working machine, comprising:

[0038] An information acquisition module, used to acquire image information of the side of the working machine and point cloud data of the side of the working machine;

[0039] An image recognition module, used to determine a classification recognition result based on the image information;

[0040] An algorithm selection module, used to determine a distance measurement algorithm based on the classification and recognition result;

[0041] The calculation module is used to use the distance measurement algorithm to process the point cloud data and determine the edge distance and running posture of the working machine, and the edge distance and running posture are used for edge control.

[0042] The present invention also provides an operating machine, comprising: an image sensor for collecting environmental images, a radar sensor for collecting point cloud data, and the edge control device of the operating machine as described above.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the edge control method of the working machine as described above are implemented.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the edge control method of the working machine as described in any one of the above are implemented.

[0045] The edge control method, device, working machine and electronic equipment of the working machine provided by the present invention can effectively improve the accuracy of the edge distance measurement results and the universality of the method by performing feature recognition on the environmental images on both sides of the working machine and selecting the corresponding distance measurement algorithm based on the feature recognition results to avoid the influence of the obstacle type on the calculation error; the edge distance and running posture can be better used for automatic edge control or auxiliary manual edge driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is one of the flow charts of the edge control method of the working machine provided by some embodiments of the present invention;

[0048] Figure 2 This is a second flow chart of a method for controlling the edge of a working machine provided by some embodiments of the present invention;

[0049] Figure 3 is a top view of a working machine provided by some embodiments of the present invention during edge-fitting operation;

[0050] Figure 4 is a schematic diagram of a method for calculating the edge distance of a working machine provided by some embodiments of the present invention;

[0051] Figure 5 is a flow chart of a method for calculating the running posture of a working machine provided by some embodiments of the present invention;

[0052] Figure 6 is a schematic structural diagram of an edge control device for a working machine provided by some embodiments of the present invention;

[0053] Figure 7is a schematic structural diagram of a working machine provided by some embodiments of the present invention;

[0054] Figure 8 It is a schematic diagram of the structure of an electronic device provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Combine the following Figure 1 The invention describes a method for controlling the edge of a working machine.

[0057] The executor of the edge control method may be a controller on the operating machine, or a control device independent of the operating machine, or a server connected to the operating machine for communication, or an operator's terminal, which may be the operator's mobile phone or computer, etc.

[0058] like Figure 1 As shown, the edge control method of the working machine provided by the embodiment of the present invention includes: step 110, step 120, step 130 and step 140.

[0059] Step 110, acquiring image information of the side surface of the working machine and point cloud data of the side surface of the working machine;

[0060] In this step, the image information is used to characterize the types of obstacles on both sides of the travel direction of the working machine.

[0061] In different scenes, the image information may contain different objects, such as curbs, railings or other obstacles.

[0062] Point cloud data is a set of vectors in a three-dimensional coordinate system.

[0063] In this step, the point cloud data is used to characterize the three-dimensional position coordinate set of points on obstacles on both sides of the working machine's travel direction. Based on the three-dimensional position coordinates, the distance between the sensor and the obstacles on both sides of the working machine can be determined.

[0064] For example, when the obstacles on both sides of the operating machine's direction of travel are curbs, the point cloud data is a set of three-dimensional coordinates corresponding to all points on the curb; or when the obstacles on both sides of the operating machine's direction of travel are railings, the point cloud data is a set of three-dimensional coordinates corresponding to all points on the railings; or when the obstacles on both sides of the operating machine's direction of travel are other obstacles other than known shape features, the point cloud data is a set of three-dimensional coordinates corresponding to all points on the corresponding other obstacles.

[0065] Under different sensor types, the type of point cloud data may be different, for example, it may include single-line lidar point cloud data, multi-line lidar point cloud data, or other solid-state radar point cloud data.

[0066] According to some embodiments of the present invention, image information may be collected by an image sensor installed on a working machine.

[0067] The image sensor can be a camera, and there can be multiple cameras, which are installed on the left and right sides of the operating machine, with the lenses tilted downward and facing outward; or installed at the front and rear ends of the operating machine, with the lenses tilted downward and facing both sides of the travel direction; or installed in various directions in front, behind, left and right of the operating machine; used to collect environmental information and ground information on both sides of the travel direction of the operating machine, to ensure that when the operating machine is very close to the obstacle, the image information of the obstacle can be collected.

[0068] Point cloud data can be collected by radar sensors installed on work machinery.

[0069] There are at least two radar sensors, installed at the front and rear ends of the operating machine respectively; or there are four radar sensors, installed at the left and right sides of the operating machine respectively, one at the front and rear of each side, for collecting the distance between the radar and both sides of the operating machine in the direction of travel.

[0070] In actual execution, the data (image information and point cloud data) collected by the above sensors are sent to the controller.

[0071] The above are only some embodiments of the present invention, and the present invention is not limited thereto. The present invention does not limit the number and installation position of the image sensor and the radar sensor, nor does it limit the type of the image sensor and the radar sensor. In actual operation, the best solution can be selected according to the specific situation.

[0072] Step 120, determining a classification recognition result based on the image information;

[0073] In this step, since the image information obtained in step 110 itself contains obstacle features, the obstacle category in the image information can be obtained through image recognition, thereby obtaining a classification recognition result.

[0074] Step 130, determining a distance measurement algorithm based on the classification and recognition result;

[0075] It should be noted that the inventors discovered the following technical problems during the research and development process: In the related technology, when performing edge control, the structural shape differences of different types of obstacles were not taken into account, and the same algorithm was used to measure the distance of all obstacles, resulting in insufficient accuracy of the edge distance and being unable to be applied to various road conditions.

[0076] During the research and development process, the inventors found that the structural shapes of different types of obstacles vary greatly. Selecting a ranging algorithm that is suitable for the obstacle type can obtain high-precision ranging results for various road conditions.

[0077] Therefore, the edge control method of the embodiment of the present invention pre-sets multiple sets of distance measurement algorithms to target various types of obstacles, thereby helping to improve the distance measurement accuracy and the universality of the method.

[0078] For example, in actual implementation, when the classification and recognition result is a curb, the ranging algorithm corresponding to the curb is selected; when the classification and recognition result is a railing, the ranging algorithm corresponding to the railing is selected; when the classification and recognition result is other obstacles, the ranging algorithm corresponding to other obstacles is selected.

[0079] Step 140: Use the distance measurement algorithm to process the point cloud data to determine the edge distance and running posture of the working machine. The edge distance and running posture are used for edge control.

[0080] In this step, the distance measurement algorithm used corresponds to the classification recognition result obtained after image recognition.

[0081] The edge distance refers to the lateral distance between the bottom side edge of the operating machinery close to the obstacle and the edge of the obstacle.

[0082] The edge distance is used to characterize the lateral distance between the operating machinery and the obstacle.

[0083] In some work scenarios where edge-to-edge operations are required, there are often higher requirements for the accuracy of the edge distance.

[0084] The running posture includes the angle between the working machine body and the obstacle boundary, which is used to characterize the angular direction of the working machine's running.

[0085] When the angle is obtuse, it means that the operating machine is about to move away from the obstacle; when the angle is acute, it means that the operating machine is about to approach the obstacle; when the angle is 0, it means that the operating machine body is parallel to the obstacle.

[0086] The edge distance and the running posture can be used together for the edge control of the working machinery.

[0087] According to some embodiments of the present invention, by inputting the edge distance and the running posture into an automatic controller used to control the running of the working machine, the automatic controller determines the distance of the working machine from the obstacle and the current running angle based on the input data, generates a control instruction based on the judgment result, and then controls the steering system to achieve automatic edge control of the working machine.

[0088] According to other embodiments of the present invention, by outputting the edge distance and the running posture in the form of numerical values, voice or images, driving assistance information can be provided to the operator.

[0089] According to the edge control method for a working machine provided by an embodiment of the present invention, environmental images on both sides of the working machine are collected and feature recognition is performed, and a corresponding distance measurement algorithm is selected based on the feature recognition result to avoid the influence of obstacle type on calculation errors, thereby effectively improving the accuracy of the measurement results and the universality of the method; and the running posture is determined by the edge distance, which can be better used for automatic edge control or to assist manual edge driving.

[0090] According to some embodiments of the present invention, step 120, determining a classification recognition result based on the image information, further includes:

[0091] The image information is input into the image recognition model, and the classification recognition result corresponding to the image information is output;

[0092] The image recognition model is a neural network model, which is trained by taking sample image information as samples and predetermined classification recognition results corresponding to the sample image information as sample labels.

[0093] In this step, the image recognition model may include a neural network for identifying input image features and outputting corresponding feature recognition results.

[0094] When training the image recognition model, sample image information and the classification recognition results corresponding to the sample image information are input, and the image recognition model is trained with a predetermined sample set to obtain a trained image recognition model for subsequent recognition applications.

[0095] For example, a sample set may include sample image information containing a curb and a label for the sample image information - curb; a sample set may include sample image information containing a railing and a label for the sample image information - railing; of course, a sample set may also include other types of samples, such as sample image information containing roadside trees.

[0096] Depending on the type of samples in the sample set, the trained image recognition model can be applied to different scenarios.

[0097] In some embodiments, the neural network in the image recognition model can be a convolutional network, and the convolutional network training process includes making a data set, defining the convolutional neural network structure, defining optimization and loss functions, network training, and structural testing and verification. The convolutional network can expand the detection types according to demand. If a new road boundary type is added, it only needs to follow the above training steps to train the convolutional network once. When in use, the classification and recognition results can be directly output after inputting the environment image, which is simple and fast.

[0098] According to the edge control method of the working machinery provided by the embodiment of the present invention, the image information is classified and recognized through the image recognition model and the classification recognition result is output, which can accurately distinguish the types of curbs, railings or other road boundaries; if a new recognition object is to be added, the training can be completed by simply training the image recognition model once according to the same training steps. The operation is simple and fast, and has good flexibility.

[0099] According to some embodiments of the present invention, before performing step 140, using a distance measurement algorithm to process point cloud data to determine the edge distance and the running posture of the working machine, the method further includes:

[0100] Set the reference coordinate system with the target point of the operating machine as the origin; convert the point cloud data to the reference coordinate system.

[0101] During the research and development process, the inventors discovered that in actual operations, depending on the needs of different working environments, different working machines or different working scenarios, the positions of sensors used to collect point cloud data installed on the working machines are often different, which results in the collected point cloud data being in different coordinate systems.

[0102] The reference coordinate system is set with the target point of the operating machine as the origin. In this step, by establishing a unified reference coordinate system, point cloud data in different coordinate systems can be quickly converted to the same coordinate system in the subsequent calculation process.

[0103] In some embodiments, the origin of the reference coordinate system can be set as the projection of the center of the working machine on the ground, the forward direction of the working machine is set as the X-axis direction, the vertical upward direction perpendicular to the road surface is set as the Z-axis, and the Y-axis direction is determined according to the right-hand rule, and this is used as the reference coordinate system. However, the present invention is not limited to this. For example, the origin of the reference coordinate system can also be set as the center of a wheel on a certain side of the working machine, or as the position of a sensor installed on the working machine, or as any other position where coordinates can be measured, and the present invention does not limit this.

[0104] After setting the reference coordinate system, the point cloud data can be converted to the reference coordinate system.

[0105] In this step, based on the reference coordinate system set in the previous step, the collected point cloud data is converted to the reference coordinate system according to the coordinate transformation relationship. This can ensure that the point cloud data collected by all sensors at different positions on the same operating machine are in the same reference coordinate system, avoiding deviations in calculation results due to inconsistent coordinate systems in subsequent calculations.

[0106] According to the edge control method of the working machinery provided by the embodiment of the present invention, the coordinate system is unified by converting the point cloud data based on the sensor coordinate system into the reference coordinate system of the working machinery, thereby avoiding deviations in the calculation results due to inconsistent coordinate systems in subsequent calculations, thereby improving the precision and accuracy of the measurement results.

[0107] Combine the following Figure 2-Figure 4 The following describes an edge control method for a working machine provided by an embodiment of the present invention.

[0108] like Figure 3 As shown, the working machine is a road roller, which performs compaction operations within the area defined by the road boundary; the road roller is equipped with a first single-line laser radar 311 and a second single-line laser radar 312, as well as a first camera 321 and a second camera 322.

[0109] The first single-line laser radar 311 and the second single-line laser radar 312 are respectively installed at the middle position of the front and rear sides of the roller, and are used to collect the lateral distance from the obstacles on both sides of the roller's moving direction; the first camera 321 and the second camera 322 are respectively installed at the front ends of the left and right sides of the roller, with the lenses facing obliquely downward and outward, and are used to collect environmental images and road surface information on both sides of the roller's moving direction.

[0110] The first single-line laser radar 311, the second single-line laser radar 312, the first camera 321, and the second camera 322 are electrically connected to the controller respectively.

[0111] like Figure 2 As shown, during the actual operation of the road roller, after acquiring image information and point cloud data through sensors, the image information is input into the image recognition model, the image data is classified and recognized based on the image recognition model, and the classification recognition result is output, and then the corresponding distance measurement algorithm is selected according to the classification recognition result.

[0112] In some embodiments, the classification and recognition result is a curb, and a distance measurement algorithm is used to process the point cloud data to determine the edge distance of the operating machinery. The specific steps are described as follows.

[0113] Continue to refer Figure 3 First, set the reference coordinate system. The origin of the reference coordinate system is the projection of the center of the roller on the ground. The reference coordinate system includes the X direction along the travel direction, the Y direction along the horizontal direction, and the Z direction along the vertical direction.

[0114] The point cloud data collected by the first single-line laser radar 311 and the second single-line laser radar 312 are converted into point cloud data under the reference system coordinates, and the point cloud data are input into the distance measurement algorithm.

[0115] According to the curb characteristics, a threshold range in the Z direction is set, such as 5 mm ≤ Z ≤ 300 mm, as a height target range, and the point cloud data is screened based on the height target range, and the point cloud data within the height target range is retained to obtain the first point cloud data;

[0116] The first point cloud data is a set of three-dimensional position coordinates of points on the curb.

[0117] Set a threshold t, cluster the first point cloud data into one category, remove the point clouds whose distance from the cluster center is greater than t, and obtain the second point cloud data;

[0118] The second point cloud data is a set of three-dimensional position coordinates of some points in the point cloud data corresponding to the curb that are closest to the installation point of the single-line laser radar.

[0119] The average value in the Y direction is solved for the second point cloud data to obtain data values ​​a1 and a2, which respectively represent the lateral distances of the installation points of the first single-line laser radar 311 and the second single-line laser radar 312 from the curb in the reference system coordinates;

[0120] Obtain the lateral distance b1 between the first single-line laser radar 311 and the front wheel of the roller, and the lateral distance b2 between the second single-line laser radar 312 and the rear wheel of the roller, and use the formulas "c1 = a1-b1, c2 = a2-b2" to calculate the first edge distance c1 and the second edge distance c2 respectively. The first edge distance represents the distance between the outer edge of the front wheel of the roller and the curb, and the second edge distance represents the distance between the outer edge of the rear wheel of the roller and the curb.

[0121] After obtaining the edge distance, according to the method of the above embodiment, using the sine function, the angle θ between the roller and the curb, that is, the running posture of the roller, can be obtained. Combined with the edge distance and running posture of the roller, the current position and posture of the roller can be accurately and clearly characterized, so as to be better used for automatic control or assisted driving.

[0122] In other embodiments, the classification and recognition result is a railing, and a distance measurement algorithm is used to process the point cloud data to determine the edge distance of the operating machinery. The specific steps are described as follows.

[0123] The point cloud data collected by the first single-line laser radar 311 and the second single-line laser radar 312 are converted into point cloud data under the reference system coordinates, and the point cloud data are input into the distance measurement algorithm.

[0124] like Figure 4 As shown, according to the railing characteristics, the upper and lower limits of the railing height are determined, and a threshold range in the Z direction is set as a height target range, such as a threshold range that can be a range between the upper and lower limits of the railing height. Based on the height target range, the point cloud data is screened, and the point cloud data within the height target range is retained to obtain the third point cloud data;

[0125] The third point cloud data is a set of three-dimensional position coordinates of points on the railing.

[0126] Continue to refer Figure 4 , set the height H of the sliding window, slide the sliding window from top to bottom along the Z direction from the upper limit of the height, slide a fixed interval distance d each time, extract the point cloud data of multiple sliding window positions, and obtain the point cloud data corresponding to each window;

[0127] In this step, the height H of the sliding window is generally set to 4 / 5 of the length of the cross-section of the railing, and the spacing distance d=1 / 4*H.

[0128] Perform a third-order polynomial fitting on the point cloud data corresponding to each window to obtain the fitting error, and select the curve function corresponding to the minimum fitting error as the curve function corresponding to the railing;

[0129] Based on the curve function corresponding to the railing, the two minimum values ​​in the Y direction are calculated, and the average value of these two minimum values ​​is the lateral distance between the installation point of the single-line laser radar and the railing in the reference coordinate system.

[0130] After the lateral distance is obtained, the edge distance can be determined based on the lateral distance. This step is the same as the step of the embodiment corresponding to the curb feature described above, and will not be described in detail here.

[0131] In some other embodiments, the classification and identification result is other obstacles, and a distance measurement algorithm is used to process the point cloud data to determine the edge distance of the operating machinery. The specific steps are described as follows.

[0132] The point cloud data collected by the first single-line laser radar 311 and the second single-line laser radar 312 are converted into point cloud data under the reference system coordinates, and the point cloud data are input into the distance measurement algorithm.

[0133] Based on the distance to the target range and the value of the point cloud data in the Y direction, multiple categories are clustered, and the point cloud data whose cluster center is closest to the operating machinery is selected as the candidate point cloud data;

[0134] The candidate point cloud data is a set of three-dimensional position coordinates of points on the obstacle closest to the operating machinery.

[0135] In this step, the target is any obstacle other than a road sign with known shape features, such as a tree, a rock, a person or other working machinery.

[0136] The candidate point cloud data obtained through this step is the point cloud data of the obstacle that is closest to the operating machine in the lateral direction.

[0137] The average value in the Y direction is solved for the candidate point cloud data, which is the lateral distance between the installation point of the single-line laser radar and the nearest obstacle.

[0138] After the lateral distance is obtained, the edge distance can be determined based on the same steps as the above two embodiments.

[0139] In some embodiments, when it is necessary to expand the road boundary categories, the image recognition model can be retrained according to the method provided in step 120 to output the corresponding road boundary classification recognition results.

[0140] According to the features corresponding to the road boundary, a threshold range in the Z direction is set as a height target range, the point cloud data is screened based on the height target range, and the point cloud data within the height target range is retained to obtain fourth point cloud data;

[0141] The fourth point cloud data is a set of three-dimensional position coordinates of points on the road boundary.

[0142] Set the height H1 of the sliding window, and within the height target range, slide the sliding window from top to bottom along the Z direction, with a fixed interval distance d1 each time, to extract the point cloud data corresponding to each window;

[0143] Perform polynomial fitting on the point cloud data corresponding to each window respectively to obtain the fitting error, and select the curve function corresponding to the minimum fitting error as the curve function corresponding to the road boundary;

[0144] Based on the curve function, the extreme value in the Y direction is calculated, and the extreme value is averaged. The average value is the lateral distance between the installation point of the single-line laser radar and the boundary of the road in the reference coordinate system.

[0145] After the lateral distance is obtained, the edge distance can be determined based on the same steps as the above three embodiments.

[0146] After obtaining the front and rear edge distances of the roller, the angle between the roller and the road, guardrail or other obstacle boundary can be obtained through the two front and rear edge distances, that is, the running posture of the roller. By inputting the edge distance and running posture into the controller or displaying them to the operator, the roller's own position and posture can be adjusted in time to perform edge compaction operations.

[0147] According to the edge control method of the working machine provided by the embodiment of the present invention, by collecting environmental images on both sides of the working machine and performing feature recognition, the corresponding distance measurement algorithm is selected based on the feature recognition result to avoid the influence of obstacle type on calculation error, effectively improve the accuracy of the algorithm, and thus improve the accuracy of the measurement result. The running posture is obtained by the edge distance to jointly display the implementation status of the working machine, and then the edge distance and the running posture are input into the controller or displayed to the operator, so as to realize the automatic edge control of the working machine, or assist the operator to adjust the position and posture of the working machine in time, so as to better perform the edge compaction operation.

[0148] According to some embodiments of the present invention, step 140, after acquiring the point cloud data, uses a distance measurement algorithm to process the point cloud data to determine the edge distance and the walking posture of the operating machine, and further includes: step 141, step 142 and step 143, such as Figure 5 shown.

[0149] During an actual operation, the operating machine may include a first sensor and a second sensor spaced apart in a front-rear direction.

[0150] The first sensor is used to collect point cloud data at the front end of the working machine, and the second sensor is used to collect point cloud data at the back end of the working machine.

[0151] Step 141, determining a first edge distance of the operating machine based on the point cloud data collected by the first sensor;

[0152] In this step, the first edge distance includes a first lateral distance between the front end of the working machine on the side close to the obstacle and the obstacle.

[0153] Based on the classification and recognition result, a corresponding distance measurement algorithm is selected, and the point cloud data collected by the first sensor is input. The first edge distance can be obtained through the distance measurement algorithm.

[0154] In some embodiments, when the classification and recognition result is a curb, the distance measurement algorithm corresponding to the curb is selected, and the point cloud data collected by the first sensor is input. Through the distance measurement algorithm, the lateral distance between the front end of the working machinery close to the curb and the curb can be obtained.

[0155] Step 142, determining a second edge distance of the operating machine based on the point cloud data collected by the second sensor;

[0156] In this step, the second edge distance includes a first lateral distance between the rear end of the working machine on the side close to the obstacle and the obstacle.

[0157] The steps for determining the second edge distance are the same as step 141 and will not be described in detail here.

[0158] Step 143: Determine the running posture based on the first edge distance and the second edge distance.

[0159] In this step, after obtaining the first edge distance and the second edge distance, the difference a between the first edge distance and the second edge distance, and the difference b between the first sensor and the second sensor along the forward direction of the working machine can be used to obtain the tangent value of the angle between the working machine body and the obstacle through the tangent function, and then the angle value can be calculated.

[0160] The traveling posture includes the angle between the working machine body and the obstacle, which is used to characterize the angle of travel of the working machine. When the angle is an obtuse angle, it means that the working machine is about to move away from the obstacle; when the angle is an acute angle, it means that the working machine is about to approach the obstacle; when the angle is 0, it means that the working machine body is parallel to the obstacle.

[0161] According to the edge control method of the working machinery provided by the embodiment of the present invention, the running posture of the working machinery is obtained by calculating the first edge distance and the second edge distance. While obtaining the distance between the working machinery and the obstacle, the angular direction of the working machinery can also be obtained. The real-time posture position of the working machinery can be presented more comprehensively and accurately, providing strong support for subsequent automatic control or manual assisted driving.

[0162] In an actual operation scenario, after completing step 140, using a distance measurement algorithm to process point cloud data and determine the edge distance and running posture of the operating machine, the edge control operation of the operating machine will be optimized based on the edge distance and running posture.

[0163] In some embodiments, a steering system of the work machine is controlled based on the edge distance and the running posture.

[0164] Among them, the edge distance represents the lateral distance between the operating machine and the obstacle, and the walking posture represents the angle between the operating machine and the boundary of the obstacle.

[0165] By inputting the edge distance and running posture into the controller of the automatic control system of the working machinery, the controller can determine the current position and posture of the working machinery. According to the working environment requirements, the controller generates corresponding control instructions to automatically adjust the steering of the working machinery to adjust the current position and posture of the working machinery so that it can operate within the specified working environment requirements.

[0166] For example, when the edge distance received by the controller is greater than the edge distance specified for the operation, the controller controls the steering system to turn to the side of the road boundary to reduce the edge distance until the edge distance meets the specified requirements; then the steering system of the operating machine continues to be adjusted through the walking posture until the body of the operating machine is kept parallel to the road boundary, and then maintains moving in this direction to perform the operation.

[0167] In other embodiments, driving assistance information is output based on the edge distance and the driving posture.

[0168] In this embodiment, the edge distance and the running posture are displayed to the operator to show the operator the current position and posture of the working machine. The operator adjusts the steering of the working machine according to the current position and posture until the working machine is adjusted to within the distance range required for the operation.

[0169] The output may be in at least one of the following forms:

[0170] First, the output can be expressed as data output.

[0171] In this embodiment, the distance value and angle value characterizing the edge distance and running posture of the working machine are directly output to the display screen interface of the operating console. The operator controls the steering system of the working machine according to the changes in the values ​​to adjust the position and posture of the working machine until the values ​​meet the requirements.

[0172] Second, the output can be in the form of image output.

[0173] In this embodiment, the real-time posture of the working machine, the boundary of surrounding obstacles and the target motion trajectory can be displayed on the image interface.

[0174] Compared with other output forms, this form can more intuitively display the current position and posture of the operating machinery, and assist operators in adjusting the steering of the operating machinery.

[0175] Third, the output can be in the form of voice output.

[0176] In this embodiment, the terminal can broadcast the boundary distance and running posture of the current operating machine by voice.

[0177] Of course, in other embodiments, the output may also be in other forms, including but not limited to signal light output, etc., which can be determined according to actual needs and is not limited in this embodiment of the present invention.

[0178] According to the edge control method of the working machine provided by the embodiment of the present invention, by outputting the edge distance and running posture of the working machine, the controller can be used to automatically control the steering system of the working machine, realize the automatic edge operation of the working machine, and then realize unmanned operation, especially in some dangerous working environments, through unmanned operation, the life safety of the operating personnel can be effectively guaranteed. In addition, it can also be used for assisted driving, by presenting a more accurate and clearer position and posture of the working machine to the operating personnel to meet the high-precision operation requirements.

[0179] The following is a description of the edge control device for the working machine provided by the present invention. The edge control device for the working machine described below and the edge control method for the working machine described above can be referenced to each other.

[0180] Figure 6 A device for controlling the edge of an operating machine is provided. Figure 6 As shown, it includes: an information acquisition module 610, an image recognition module 620, an algorithm selection module 630 and a calculation module 640.

[0181] An information acquisition module 610 is used to acquire image information of the side of the working machine and point cloud data of the side of the working machine;

[0182] An image recognition module 620 is used to determine a classification recognition result based on the image information;

[0183] An algorithm selection module 630 is used to determine a distance measurement algorithm based on the classification and recognition result;

[0184] The calculation module 640 is used to use the distance measurement algorithm to process the point cloud data and determine the edge distance and running posture of the working machine. The edge distance and running posture are used for edge control.

[0185] In some embodiments, the working machine includes a first sensor and a second sensor spaced apart in a front-to-rear direction;

[0186] Use distance measurement algorithms to process point cloud data and determine the edge distance and walking posture of the operating machinery, including:

[0187] Determine a first edge distance of the operating machine based on the point cloud data collected by the first sensor;

[0188] Determine a second edge distance of the operating machine based on the point cloud data collected by the second sensor;

[0189] The running posture is determined based on the first edge distance and the second edge distance.

[0190] In some embodiments, before using the distance measurement algorithm to process the point cloud data to determine the edge distance and the walking posture of the working machine, the method further includes:

[0191] Set the reference coordinate system with the target point of the operating machine as the origin;

[0192] Transform point cloud data into a reference coordinate system.

[0193] In some embodiments, the reference coordinate system includes an X direction along the traveling direction, a Y direction along the lateral direction, and a Z direction along the vertical direction;

[0194] When the classification result is a curb, a distance measurement algorithm is used to process the point cloud data and determine the edge distance of the operating machine, including:

[0195] Based on the height target range corresponding to the classification recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain the first point cloud data;

[0196] Clustering the first point cloud data into one category, removing point clouds beyond a first target distance from a cluster center, and obtaining second point cloud data;

[0197] Determine the edge distance based on the second point cloud data;

[0198] Alternatively, when the classification result is a railing, a distance measurement algorithm is used to process the point cloud data to determine the edge distance of the operating machine, including:

[0199] Based on the height target range corresponding to the classification recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain third point cloud data;

[0200] Extracting point cloud data based on the Z direction on the third point cloud data, fitting the extracted point cloud data, and obtaining a point cloud curve function;

[0201] Based on the point cloud curve function, determine the edge distance;

[0202] Alternatively, when the classification and recognition result is other, a distance measurement algorithm is used to process the point cloud data to determine the edge distance of the operating machine, including:

[0203] Based on the distance to the target range and the value of the point cloud data in the Y direction, multiple categories of clustering are performed, and the point cloud whose cluster center is closest to the operating machinery is selected as the candidate point cloud data;

[0204] Based on the candidate point cloud data, the edge distance is determined.

[0205] In some embodiments, determining a classification recognition result based on the image information includes:

[0206] The image information is input into the image recognition model, and the classification recognition result corresponding to the image information is output;

[0207] The image recognition model is a neural network model, which is trained by taking sample image information as samples and predetermined classification recognition results corresponding to the sample image information as sample labels.

[0208] In some embodiments, after using the distance measurement algorithm to process the point cloud data and determine the edge distance and the walking posture of the working machine, the method further includes:

[0209] Control the steering system of the operating machinery based on the edge distance and running posture;

[0210] or,

[0211] Based on the edge distance and driving posture, driving assistance information is output. The auxiliary information can be prompted in the form of voice or image. The driver can adjust the vehicle steering wheel according to the prompt to ensure that the vehicle's direction of travel is approximately parallel to the road boundary.

[0212] According to the edge control device of the working machine provided by the embodiment of the present invention, the environmental images on both sides of the working machine are collected by the information acquisition module 610, the environmental images are identified by the image recognition module 620, and the corresponding distance measurement algorithm is selected based on the feature recognition result by the algorithm selection module 630 to avoid the influence of the obstacle type on the calculation error, effectively improve the accuracy of the algorithm, and thus improve the accuracy of the measurement result. Finally, the calculation module 640 processes the point cloud data to determine the edge distance and running posture of the working machine, which are used to jointly represent the real-time working status of the working machine. By inputting the edge distance and running posture into the controller or displaying them to the operator, the roller's own position and posture are adjusted in time to better perform the edge compaction operation.

[0213] The working machine provided by the present invention is described below.

[0214] The working machine according to the embodiment of the present invention may be a road roller, a road sweeper, or a road cold spray marking machine.

[0215] The working machine is provided with an image sensor, a radar sensor and an edge control device of the working machine.

[0216] The image sensor is used to collect environmental images on both sides of the working machine in the direction of travel, and the radar sensor is used to collect point cloud data. The image sensor and the radar sensor are installed on the working machine respectively, and the number of radar sensors is at least two, which are used to collect point cloud data at the front and rear ends of the working machine respectively.

[0217] like Figure 7 As shown, the working machine is a road roller, and the road roller is provided with a first camera 711, a second camera 712, a first radar 721, a second radar 722 and the edge control device of the working machine as described above.

[0218] In some embodiments, the first camera 711 and the second camera 712 collect environmental images and road surface information on both sides of the roller's moving direction, and input them into the roller's edge control device to perform feature extraction to obtain classification and recognition results, and determine the distance measurement algorithm based on the classification and recognition results.

[0219] The point cloud data collected by the first radar 721 and the second radar 722 are input into the roller's edge control device to convert them into point cloud data under the roller's reference system coordinates, and the point cloud data are input into the corresponding distance measurement algorithm. The distance measurement algorithm is used to process the point cloud data to determine the roller's edge distance and running posture.

[0220] By inputting the edge distance and running posture into the controller used to automatically control the roller's running, the roller can be automatically controlled to adjust its own position and posture to achieve automatic edge control. Alternatively, the edge distance and running posture can be displayed to the operator to assist the operator in timely adjusting the position and posture of the roller to achieve high-precision edge control.

[0221] According to the working machinery provided by the embodiment of the present invention, image sensors and radar sensors are arranged on the working machinery to collect environmental images and point cloud data on both sides of the working machinery in real time; the working machinery edge control method as described above is arranged on the working machinery to perform feature recognition on the environmental image, and a corresponding distance measurement algorithm is selected based on the feature recognition result to avoid the influence of obstacle type on calculation error, effectively improve the accuracy of the algorithm, thereby improving the accuracy of the edge distance measurement result, better serving the unmanned automatic operation of the working machinery, or better assisting the operation personnel in their work.

[0222] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the edge control method of the working machine, the method comprising: obtaining the image information of the side of the working machine and the point cloud data of the side of the working machine; determining the classification and recognition result based on the image information; determining the distance measurement algorithm based on the classification and recognition result; using the distance measurement algorithm, processing the point cloud data, determining the edge distance and the running posture of the working machine, the edge distance and the running posture are used for edge control.

[0223] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0224] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the edge control method of the working machinery provided by the above-mentioned methods, and the method includes: obtaining image information of the side of the working machinery and point cloud data of the side of the working machinery; based on the image information, determining a classification and recognition result; based on the classification and recognition result, determining a ranging algorithm; using the ranging algorithm, processing the point cloud data, and determining the edge distance and running posture of the working machinery, and the edge distance and the running posture are used for edge control.

[0225] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the edge control method of the working machine provided by the above-mentioned methods. The method includes: acquiring image information of the side of the working machine and point cloud data of the side of the working machine; determining a classification and recognition result based on the image information; determining a ranging algorithm based on the classification and recognition result; using the ranging algorithm to process the point cloud data to determine the edge distance and running posture of the working machine, and the edge distance and the running posture are used for edge control.

[0226] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0227] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling the edge of an operating machine, characterized in that: include: Acquire image information of the side surface of the working machine and point cloud data of the side surface of the working machine; Based on the image information, determining a classification and recognition result; Determining a distance measurement algorithm based on the classification and recognition result; Using the distance measurement algorithm, processing the point cloud data, and determining the edge distance and running posture of the working machine, wherein the edge distance and the running posture are used for edge control; Before using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine, the method further includes: Taking the target point of the working machine as the origin, setting a reference coordinate system; Converting the point cloud data to the reference coordinate system; The reference coordinate system includes an X direction along the traveling direction, a Y direction along the horizontal direction, and a Z direction along the vertical direction; In the case where the classification and recognition result is a curb, based on the height target range corresponding to the classification and recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain first point cloud data; Clustering the first point cloud data into one category, removing point clouds beyond a first target distance from a cluster center, and obtaining second point cloud data; Determining the edge distance based on the second point cloud data; In the case where the classification and recognition result is a railing, based on the height target range corresponding to the classification and recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain third point cloud data; Extracting point cloud data based on the Z direction from the third point cloud data, and fitting the extracted point cloud data to obtain a point cloud curve function; Determining the edge distance based on the point cloud curve function; In the case where the classification and recognition result is other, clustering is performed into multiple categories based on the distance target range and the value of the point cloud data in the Y direction, and the point cloud whose cluster center is closest to the operating machine is selected as the candidate point cloud data; The edge distance is determined based on the candidate point cloud data.

2. The method for controlling the edge of a working machine according to claim 1, characterized in that: The working machine includes a first sensor and a second sensor arranged at intervals in the front-rear direction; The method of using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine includes: Determining a first edge distance of the operating machine based on the point cloud data collected by the first sensor; Determining a second edge distance of the operating machine based on the point cloud data collected by the second sensor; The walking posture is determined based on the first edge distance and the second edge distance.

3. The method for controlling the edge of a working machine according to any one of claims 1 to 2, characterized in that: The step of determining a classification and recognition result based on the image information includes: The image information is input into an image recognition model, and the classification recognition result corresponding to the image information is output; wherein The image recognition model is a neural network model, which is trained using sample image information as samples and predetermined classification recognition results corresponding to the sample image information as sample labels.

4. The method for controlling the edge of a working machine according to any one of claims 1 to 2, characterized in that: After using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine, the method further includes: Based on the edge distance and the running posture, controlling the steering system of the working machine; or, Based on the edge distance and the running posture, driving assistance information is output.

5. A control device for the edge of an operating machine, characterized in that: include: An information acquisition module, used to acquire image information of the side of the working machine and point cloud data of the side of the working machine; An image recognition module, used to determine a classification recognition result based on the image information; An algorithm selection module, used to determine a distance measurement algorithm based on the classification and recognition result; A calculation module, used to use the distance measurement algorithm to process the point cloud data, and determine the edge distance and running posture of the working machine, wherein the edge distance and the running posture are used for edge control; Before using the distance measurement algorithm to process the point cloud data and determine the edge distance and the running posture of the working machine, the method further includes: Taking the target point of the working machine as the origin, setting a reference coordinate system; Converting the point cloud data to the reference coordinate system; The reference coordinate system includes an X direction along the traveling direction, a Y direction along the horizontal direction, and a Z direction along the vertical direction; In the case where the classification and recognition result is a curb, based on the height target range corresponding to the classification and recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain first point cloud data; Clustering the first point cloud data into one category, removing point clouds beyond a first target distance from a cluster center, and obtaining second point cloud data; Determining the edge distance based on the second point cloud data; In the case where the classification and recognition result is a railing, based on the height target range corresponding to the classification and recognition result and the value of the point cloud data in the Z direction, the point cloud data is filtered to obtain third point cloud data; Extracting point cloud data based on the Z direction from the third point cloud data, and fitting the extracted point cloud data to obtain a point cloud curve function; Determining the edge distance based on the point cloud curve function; In the case where the classification and recognition result is other, clustering is performed into multiple categories based on the distance target range and the value of the point cloud data in the Y direction, and the point cloud whose cluster center is closest to the operating machine is selected as the candidate point cloud data; The edge distance is determined based on the candidate point cloud data.

6. A working machine, characterized in that: include: An image sensor for collecting environmental images, a radar sensor for collecting point cloud data, and an edge control device for a working machine as claimed in claim 5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the edge control method of the working machine as claimed in any one of claims 1 to 4 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the edge control method of the working machine as claimed in any one of claims 1 to 4 are implemented.

Images