Navigation method and device of pavement machinery and pavement machinery

Abstract

The invention provides a navigation method and device for a pavement machine and the pavement machine, and the navigation method for the pavement machine comprises the following steps: acquiring continuous scene images through an image sensor, the continuous scene images comprising a plurality of single-frame scene images; determining a first pixel point set of the marker in the single-frame scene image according to a deep learning network model; screening the first pixel point set to form a second pixel point set; performing linear optimization on the second pixel point set to form a third pixel point set; performing straight line fitting on the third pixel point set to generate straight line parameters; combining the straight line parameter with the single-frame scene image to generate world coordinate information of the marker; and generating a navigation path according to the world coordinate information in the plurality of single-frame scene images, and controlling the pavement machinery to run along the navigation path. Through the scheme provided by the invention, the paver can accurately know the position of the paver relative to the road surface marker, and the anti-interference capability can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection, and in particular to a navigation method and device for a road surface machine and the road surface machine. Background Art

[0002] When driving road machinery such as pavers, the construction site is relatively complex and the time spent in a single scene is long. The image processing for assisted driving often fails in detection, resulting in poor following effects for assisted driving and reduced control stability of the road machinery. Summary of the invention

[0003] The present application discloses a navigation method for a road surface machine, a navigation device for a road surface machine and a road surface machine, so as to solve or improve the problem in the related art that the paver has a weak anti-interference ability during the auxiliary driving process.

[0004] In order to solve the above problems, the first aspect of the present invention proposes a navigation method for road machinery, including: acquiring continuous scene images through an image sensor, the continuous scene images including multiple single-frame scene images; determining a first pixel point set of a marker in the single-frame scene image according to a deep learning network model; screening the first pixel point set to form a second pixel point set; performing linear optimization on the second pixel point set to form a third pixel point set; performing straight line fitting on the third pixel point set to generate straight line parameters; combining the straight line parameters with the single-frame scene image to generate world coordinate information of the marker; generating a navigation path according to the world coordinate information in the multiple single-frame scene images, and controlling the road machinery to travel along the navigation path.

[0005] According to the method provided by the present invention, the accuracy and stability of machine vision in complex environments can be improved, especially in the application of automated construction equipment such as pavers. Specifically, by acquiring continuous scene images, it is ensured that there is sufficient data for dynamic analysis and processing, providing a continuous image data stream, and providing a basis for stable detection in dynamic environments.

[0006] Based on some embodiments, optionally, a first pixel point set of a marker in a single-frame scene image is determined according to a deep learning network model, including: inputting the single-frame scene image into the deep learning network model; identifying the marker in the single-frame scene image through the deep learning network model, and obtaining a first pixel point set corresponding to the marker.

[0007] Based on some embodiments, optionally, the step of filtering the first pixel point set to form the second pixel point set includes: determining a filtering threshold of the first pixel point set; and filtering the pixel points in the Y-axis direction in the single-frame scene image according to the filtering threshold to form the second pixel point set.

[0008] Based on some embodiments, optionally, the first pixel point set is screened to form a second pixel point set, including: determining the world coordinates of a reference object on the road surface machinery, wherein the relative position of the reference object and the image sensor is fixed; converting the world coordinates of the reference object into image coordinates of the reference object according to external parameters of the image sensor; and determining the first pixel point set in the area of ​​interest as the second pixel point set according to the image coordinates of the reference object and the image coordinates of the marker.

[0009] Based on some embodiments, optionally, the second pixel point set is optimized for linearity to form a third pixel point set, including: determining the first pixel point of the second pixel point set in the X-axis direction in the single-frame scene image; performing variance statistics on adjacent first pixel points to obtain adjacent pixel point variances; determining the linearity of the second pixel point set based on the adjacent pixel point variances; and optimizing the linearity of the second pixel point set based on the linearity of the second pixel point set to form a third pixel point set.

[0010] On the basis of some embodiments, optionally, the linearity of the second set of pixel points is determined according to the variance of adjacent pixel points, including: setting a first variance threshold and a second variance threshold, the first variance threshold is greater than the second variance threshold; when the variance of adjacent pixel points is greater than or equal to the first variance threshold, the linearity of the corresponding second set of pixel points is determined to be unstable; when the variance of adjacent pixel points is less than or equal to the second variance threshold, the linearity of the corresponding second set of pixel points is determined to be stable; when the variance of adjacent pixel points is less than the first variance threshold and greater than the second variance threshold, the linearity of the corresponding second set of pixel points is determined to be relatively stable.

[0011] On the basis of some embodiments, optionally, the linearity of the second pixel point set is optimized according to the linearity of the second pixel point set to form a third pixel point set, including: when the linearity of the second pixel point set is unstable, determining the scene images of a first number of frames before the current frame, correcting the first pixel point set of the scene image of the current frame according to the mean of the linear parameters of the scene images of the first number of frames, denoising the scene image of the current frame, determining the region of interest in the denoised scene image, and regenerating the first pixel point set according to the deep learning network model and the region of interest; when the linearity of the second pixel point set is stable, performing sliding average filtering on the second pixel point set to generate a third pixel point set. Three pixel point sets; when the linearity of the second pixel point set is relatively stable, the second pixel point set of the scene image of the current frame is matched with the second pixel point set of the scene image of the previous frame; the relative distance difference between each pixel point of the scene image of the current frame and the matched pixel point of the scene image of the previous frame is calculated, and the second pixel point corresponding to the unstable linearity in the scene image of the current frame is determined according to the relative distance difference; two third pixel points adjacent to the second pixel point and with stable linearity are determined, and the mean pixel point is determined according to the average value of the two third pixel points; the second pixel point is corrected according to the mean pixel point, and the corrected second pixel point is subjected to sliding average filtering to generate a third pixel point set.

[0012] On the basis of some embodiments, the straight line parameters are optionally combined with a single-frame scene image to generate the world coordinate information of the marker, including: obtaining the external parameters of the image sensor; performing an inverse perspective transformation on the third pixel point set according to the external parameters of the image sensor to generate the world coordinate information of the marker; determining the scene image of a second number of frames before the current frame; determining calibration information according to the scene image of the second number of frames before the current frame, the calibration information being the average of the world coordinate information of the marker in the scene image of the second number of frames before the current frame; determining according to the world coordinate information of the marker in the current frame and the calibration information Relative distance difference; when the relative distance difference is less than or equal to a preset threshold, the world coordinate information of the marker in the current frame is used as the world coordinate information of the marker; when the relative distance difference is greater than the preset threshold, the first pixel point set of the scene image of the current frame is re-determined according to the average value of the straight line parameters of the scene images of the third number of frames before the current frame, the scene image of the current frame is denoised, the region of interest in the denoised scene image is determined, and the first pixel point set is regenerated according to the deep learning network model and the region of interest, and new straight line parameters are generated; the new straight line parameters are combined with the single-frame scene image to generate the world coordinate information of the marker.

[0013] The second aspect of the present invention proposes a navigation device for a road machinery, comprising: an image acquisition module, used to acquire continuous scene images through an image sensor, the continuous scene images including multiple single-frame scene images; a marker detection module, used to determine a first pixel point set in a single-frame scene image according to a deep learning network model; a straight line parameter generation module, used to screen the first pixel point set to form a second pixel point set; perform linear optimization on the second pixel point set to form a third pixel point set; perform straight line fitting on the third pixel point set to generate straight line parameters; a coordinate generation module, used to combine the straight line parameters with the single-frame scene image to generate world coordinate information of the marker; a control module, used to generate a navigation path according to the world coordinate information in multiple single-frame scene images, and control the road machinery to travel along the navigation path.

[0014] A third aspect of the present invention provides a road machine, comprising a fuselage structure, on which an image sensor and a reference object are provided; the navigation device of the above-mentioned road machine is provided on the fuselage structure.

[0015] The road machine provided according to the present invention includes a body structure and a navigation device, wherein an image sensor and a reference object are arranged on the body structure, the image sensor is used to acquire a scene image, and the reference object is used to automatically capture an area of ​​interest.

[0016] Among them, since the road surface machinery includes the navigation device of the road surface machinery mentioned above, the technical effects of the navigation device of the road surface machinery mentioned above are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0021] Figure 5 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0022] Figure 6 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0023] Figure 7 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0024] Figure 8 A schematic structural diagram of a navigation device for a road surface machine according to an embodiment of the present application is shown;

[0025] Fig. 9 A schematic diagram showing the hardware structure of an electronic device according to an embodiment of the present application is shown;

[0026] Fig.10 A schematic flow chart of a navigation method for a road surface machine according to an embodiment of the present application is shown;

[0027] Fig.11 A schematic diagram showing a second pixel point set according to an embodiment of the present application is shown;

[0028] Fig.12 A schematic diagram showing a second pixel point set according to an embodiment of the present application is shown;

[0029] Fig.13 A schematic diagram showing a second pixel point set according to an embodiment of the present application is shown;

[0030] Fig.14 A schematic diagram showing a third pixel point set according to an embodiment of the present application is shown;

[0031] Fig.15 A schematic diagram showing a third pixel point set according to an embodiment of the present application is shown;

[0032] Fig.16 A schematic diagram showing straight line parameters and a navigation path of a paving machine according to an embodiment of the present application is shown;

[0033] Fig.17 A schematic structural diagram of a road surface machine according to an embodiment of the present application is shown.

[0034] in, Figures 8 to 17 The corresponding relationship between the reference numerals and the component names is as follows:

[0035] 102: second pixel point set; 1042: second pixel point; 1044: third pixel point; 1046: mean pixel point; 106: third pixel point set; 108: straight line parameter; 110: navigation path; 900: navigation device of road machinery; 901: image acquisition module; 902: marker detection module; 903: straight line parameter generation module; 904: coordinate generation module; 905: control module; 100: electronic device; 1110: processor; 1109: memory; 200: road machinery; 201: body structure; 202: image sensor; 204: reference object. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned purposes, features and advantages of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the embodiments of the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0038] Refer to the following Figures 1 to 17 A navigation method, device and road surface machinery provided by some embodiments of the present invention are described in detail.

[0039] This application proposes a navigation method for road machinery, such as Figure 1 As shown, the navigation method of the road machine in the embodiment of the present application includes the following steps:

[0040] Step S102: acquiring continuous scene images through an image sensor; the continuous scene images include a plurality of single-frame scene images;

[0041] Step S104: determining a first pixel point set of a marker in a single-frame scene image according to a deep learning network model;

[0042] Step S1062: screening the first pixel point set to form a second pixel point set;

[0043] Step S1064: performing linear optimization on the second pixel point set to form a third pixel point set;

[0044] Step S1066: performing straight line fitting on the third pixel point set to generate straight line parameters;

[0045] Step S108: combining the straight line parameters with the single-frame scene image to generate the world coordinate information of the marker;

[0046] Step S110: Generate a navigation path according to the world coordinate information in the multiple single-frame scene images, and control the road machine to travel along the navigation path.

[0047] In this embodiment, the accuracy and stability of machine vision in complex environments can be improved, especially in the application of automated construction equipment such as pavers. Specifically, by acquiring continuous scene images, ensuring that there is sufficient data for dynamic analysis and processing, providing a continuous image data stream, and providing a basis for stable detection in dynamic environments.

[0048] On this basis, by detecting the markers in a single-frame scene image, identifying and locating the markers in each frame image, such as curbs or white lime lines, a first pixel point set corresponding to the markers is generated, and the first pixel point set is fitted. Specifically, a mathematical fitting method (such as the least squares method) can be used to fit a straight line to the detected pixel point set.

[0049] The line parameters are extracted from the pixel point set, and the position and direction of the marker in the image can be determined based on the line parameters. Finally, the line parameters are combined with the single-frame scene image to generate the world coordinate information of the marker. It can be understood that the generation of a single-frame scene image can be determined based on the internal and external parameters of the camera or other image acquisition device, and the two-dimensional image coordinates can be converted into three-dimensional world coordinates, thereby providing the specific position of the marker in the real world.

[0050] Among them, the acquisition of continuous images ensures that the system can adapt to dynamically changing environments, such as the movement of the paver under different road conditions. The detection of single-frame images allows the system to capture the state of the marker at every moment. The fitting of the pixel point set is a key step that directly affects the accuracy of the line parameters and thus the final world coordinate information. The generation of world coordinates as the ultimate goal enables the paver to accurately know its position relative to the road marker, thereby achieving precise control.

[0051] It should be added that according to the navigation method of the road machinery provided in this embodiment, the accuracy and stability of the marker detection are effectively improved by preprocessing, filtering and multiple reasoning of the image. In addition, the accuracy of the straight line fitting is improved by eliminating abnormal detection points. The automatic region of interest extraction (i.e., Region of Interest, ROI) technology and the secondary reasoning mechanism improve the generalization ability of the detection system in various complex environments. Calibration and comparison between consecutive frames can be used to ensure the positioning accuracy of the paver during long-term automatic driving.

[0052] In a specific embodiment, the intrinsic and extrinsic parameters of the sensor are obtained through calibration, distortion correction is performed, the accuracy of the image is ensured, image distortion is eliminated, and accurate image data is provided for subsequent processing.

[0053] The deep learning model is used to detect the target of the preprocessed image, obtain the detection points, accurately identify the markers, and provide the detection point data. In addition, the region of interest (ROI) can be extracted to reduce interference. The ROI can be determined by manual setting or automatic calculation to reduce background interference and improve the accuracy of the detection points.

[0054] By counting the pixel differences of the detection points in the X-axis direction, the linearity of the points is determined to ensure the uniformity of the distribution of the detection points and eliminate abnormal points.

[0055] By comparing the detection point positions of the previous and next frames, using Gaussian distribution or other methods to eliminate jump points, linear interpolation is performed to remove abnormal detection points to maintain the continuity and stability of the detection results.

[0056] Perform straight line fitting on the filtered detection points to obtain straight line parameters, provide accurate straight line parameters, and provide basic data for subsequent steps.

[0057] Finally, the camera extrinsic parameters are used to perform inverse perspective transformation, converting the image coordinates into world coordinates to obtain the actual position information of the marker and improve the positioning accuracy.

[0058] The distance results of the stable frame are saved in a container, the mean is calculated as the calibration value, and compared with the current frame result to determine the stability of the detection result, ensure the continuity and stability of the detection result, and remove abnormal data in time.

[0059] In addition, for the frames that failed to be detected, new ROI areas were generated using stable frame parameters, and detection and filtering were performed again after removing interference to improve the success rate of detection, especially in complex environments.

[0060] Perform sliding mean processing on the test results that meet the requirements to reduce output value fluctuations, smooth the test results, and further improve stability.

[0061] The lateral distance, angle, status position and other information of the detection marker are sent to the control module to ensure that the paver obtains accurate positioning data during automatic driving and supports automatic driving control.

[0062] Specifically, scene images are continuously acquired to provide continuous detection data, support calibration and comparison between consecutive frames, and improve overall detection stability.

[0063] Then, by detecting the markers in the single-frame scene image, the markers in the single-frame image are detected to obtain the first pixel point set, provide single-frame detection data, and then by fitting the first pixel point set, the detected pixel point set is fitted with a straight line to generate straight line parameters to support position and direction calculation.

[0064] Finally, the straight line parameters are combined with the single-frame scene image to generate the world coordinate information of the marker, provide the accurate position information of the marker, and support automatic driving control.

[0065] like Figure 2 As shown, Figure 2 A schematic flow chart of the navigation method for a road surface machine of the present application is shown.

[0066] The method of acquiring continuous scene images through an image sensor includes:

[0067] Step S1022: acquiring a plurality of continuous first images through an image sensor;

[0068] Step S1024: performing distortion correction on the first image according to internal parameters of the image sensor to generate a single-frame scene image;

[0069] Step S1026: Determine continuous scene images according to the time sequence and the multiple single-frame scene images.

[0070] In this embodiment, when acquiring a scene image, a plurality of continuous first images are first acquired through an image sensor, and an image sensor (such as a camera) is used to capture continuous images of the environment in real time to obtain a series of image data in a time series. The first image is subjected to distortion correction according to internal parameters of the image sensor, and the captured original image is subjected to distortion correction using known internal parameters of the sensor (such as focal length, principal point coordinates, etc.) to eliminate image distortion caused by the optical characteristics of the lens, improve image quality, and provide accurate image data for subsequent processing.

[0071] The image after distortion correction becomes a single-frame scene image, providing an accurate visual snapshot of the environmental state at each moment. Finally, based on the time series and multiple single-frame scene images, continuous scene images are determined, and multiple single-frame scene images are arranged in chronological order, that is, according to the time series, to form a continuous scene image, and a time-varying environment model is constructed, which provides a basis for dynamic target tracking, motion estimation and scene understanding.

[0072] Distortion correction is equivalent to preprocessing the image to ensure the accuracy of the image data. The generation of a single-frame scene image provides clear and accurate visual information for each time point.

[0073] Among them, according to the deep learning network model, determining the first pixel point set of the marker in the single-frame scene image includes: inputting the single-frame scene image into the deep learning network model; identifying the marker in the single-frame scene image through the deep learning network model, and obtaining the first pixel point set corresponding to the marker.

[0074] When generating the first pixel point set, the distortion-corrected single-frame scene image is first used as input data and sent to the deep learning network for processing. The deep learning network can perform feature extraction and pattern recognition on the single-frame scene image. The deep learning network uses its trained model parameters to analyze the input single-frame scene image and identify specific markers in the image, such as road edges, markings, construction signs, etc. Finally, the deep learning network outputs the recognition result, which is usually the bounding box or key point of the marker on the image, and obtains the position information of the marker on the image. This information is represented in the form of pixels, providing basic data for further image analysis and processing.

[0075] Among them, according to the specific requirements of the task, a suitable deep learning architecture is selected, such as a convolutional neural network (CNN). The deep learning model needs to be trained on a large amount of labeled data to learn the characteristics and patterns of the markers. The network extracts image features through multi-layer convolution and pooling operations. These features are crucial for the recognition of markers.

[0076] The recognition results usually include the bounding box or key points of the marker, and this information can be directly used for subsequent image analysis, such as straight line fitting, shape analysis, etc.

[0077] When fitting the first pixel point set, the first pixel point set is first screened, and abnormal points that may be caused by noise or misidentification are removed from the original pixel point set, and the points that are most likely to be part of the marker are retained to form a second pixel point set, thereby improving the quality of the point set. The second pixel point set is then optimized for linearity, and the point set is further optimized, which may include removing outliers that are far away from the trend of other point sets, or identifying and strengthening straight line features through mathematical methods of random sample consensus algorithms (such as Random Sample Consensus, RANSAC algorithm), thereby enhancing the linearity of the point set and reducing the interference of non-linear features to form a third pixel point set, thereby making the straight line fitting more accurate.

[0078] Finally, a straight line is fitted to the third pixel point set to generate straight line parameters. A fitting algorithm (such as the least squares method) is used to fit the optimized point set to determine the mathematical expression of the straight line. The parameters describing the straight line, such as the slope and intercept, are generated. These parameters can be used for further analysis and application, such as calculating the actual position of the marker or its relationship with other straight lines.

[0079] The specific screening process may be based on the distribution characteristics of the pixel points, such as distance measurement, directional consistency, etc. When optimizing the linearity, an iterative method may be used to gradually eliminate bad data points that affect the linear fitting.

[0080] Through the above scheme, the system can accurately extract straight line features from the image and convert them into useful parameters, providing key information for the navigation and control of automated equipment such as pavers.

[0081] Among them, step S1062 can be implemented through the following steps: determining a screening threshold of the first pixel point set; and screening the pixel points in the Y-axis direction in the single-frame scene image according to the screening threshold to form a second pixel point set.

[0082] When screening the original first pixel set, one or more screening thresholds are set to evaluate whether the pixel is part of the marker or an abnormal point caused by noise or misidentification, which helps to distinguish valid pixels from invalid pixels. According to the screening threshold, the pixels in the Y-axis direction of the single-frame scene image are screened, and those pixels that meet specific conditions are retained. Through the screening in the Y-axis direction, those points with abnormal distribution in the vertical direction can be removed, thereby eliminating interference that does not belong to the straight line feature.

[0083] The setting of the screening threshold can be based on a variety of factors, such as the density of pixel points, the consistency of distribution, the proximity to known straight line features, etc. The Y-axis direction is selected for screening because the straight line features of the markers are easier to identify in the Y-axis direction. Of course, there is also a relationship that the driving direction of the paver is parallel or approximately parallel to the Y-axis.

[0084] Among them, possible screening algorithms include statistical-based methods (such as standard deviation thresholds), clustering algorithms, or model-based methods.

[0085] Among them, for step S1062, if Figure 3 As shown, it can be achieved through the following steps: Step S10622: determine the world coordinates of the reference object on the road machinery, and the relative position of the reference object and the image sensor is fixed; Step S10624: convert the world coordinates of the reference object into the image coordinates of the reference object according to the external parameters of the image sensor; Step S10626: determine the first pixel point set in the area of ​​interest as the second pixel point set according to the image coordinates of the reference object and the image coordinates of the marker.

[0086] When screening the first pixel point set, first select or define a reference object as a benchmark for evaluating whether the pixel point belongs to the marker. The reference object can be a component of the vehicle body, and the relative position of this component and the image sensor is fixed. By acquiring the external parameters of the image sensor, the world coordinates of the reference object can be converted into image coordinates, and then the intersection of the image coordinates of the reference object and the image coordinates of the marker is used to automatically intercept the point set in the area of ​​interest, thereby serving as the second pixel point set.

[0087] For example, in the case of a paver navigation, the reference object may be a component on the vehicle, such as a guide rod. The relative position evaluation includes calculating the distance from the pixel to the reference object, the angle difference, or whether it is on the expected path of the reference object. The screening criteria can be based on statistical analysis, such as mean, median, standard deviation, etc., or based on geometric analysis, such as straightness, parallelism, etc.

[0088] like Figure 4 As shown, Figure 4 A further limited process for step S1064 is shown. Specifically, the linearity optimization of the second pixel point set is performed to form a third pixel point set, including: step S10642: determining the first pixel point of the second pixel point set in the X-axis direction in the single-frame scene image; step S10644: performing variance statistics on the adjacent first pixel points to obtain the variance of the adjacent pixel points; step S10646: determining the linearity of the second pixel point set according to the variance of the adjacent pixel points; step S10648: optimizing the linearity of the second pixel point set according to the linearity of the second pixel point set to form a third pixel point set.

[0089] When optimizing the second pixel point set, first determine the first pixel point of the second pixel point set in the X-axis direction in the single-frame scene image, and identify the pixel points in the X-axis direction from the second pixel point set. The first pixel point located in the X-axis direction helps to simplify the linearity analysis, because a straight line is usually represented as a horizontal or vertical point set in an image. By performing variance statistics on adjacent first pixel points, the variance of adjacent pixel points is obtained. Variance statistics provides a method for quantifying the compactness of the point set distribution. Low variance means that the point set is evenly distributed in the X-axis direction. Then, the linearity of the second pixel point set is determined based on the variance of adjacent pixel points, that is, the variance statistics result is used to evaluate whether the second pixel point set is close to a straight line. If the variance is lower than a certain threshold, it can be considered that the point set has good linearity; if the variance is high, the point set may contain non-linear features or noise.

[0090] Finally, the linearity of the second pixel point set is optimized according to its linearity to form a third pixel point set. An optimization algorithm, such as the RANSAC (Random Sample Consensus) algorithm, is applied to enhance the straight line features and remove outliers. Through optimization, a purer and more linear third pixel point set is obtained, which provides more accurate data for straight line fitting.

[0091] The setting of the variance threshold needs to be determined according to the specific application scenario and image quality to balance the linearity and the integrity of the point set. In addition to RANSAC, other algorithms can also be used, such as LTS (Line Test and Sample) or optimization methods based on machine learning.

[0092] Through the above scheme, a third pixel point set with high linearity can be effectively extracted from the second pixel point set, providing a more accurate and reliable data basis for the final straight line fitting.

[0093] Among them, determining the linearity of the second pixel point set according to the variance of adjacent pixel points includes: setting a first variance threshold and a second variance threshold, the first variance threshold is greater than the second variance threshold; when the variance of adjacent pixel points is greater than or equal to the first variance threshold, the linearity of the corresponding second pixel point set is determined to be unstable; when the variance of adjacent pixel points is less than or equal to the second variance threshold, the linearity of the corresponding second pixel point set is determined to be stable; when the variance of adjacent pixel points is less than the first variance threshold and greater than the second variance threshold, the linearity of the corresponding second pixel point set is determined to be relatively stable.

[0094] When determining linearity, the first variance threshold and the second variance threshold are first set to define two thresholds for distinguishing the linearity state of the point set. The first variance threshold is usually set higher, while the second variance threshold is lower. Through these two thresholds, the linearity of the point set can be classified to provide a basis for subsequent linearity optimization. Specifically, when the variance of adjacent pixel points is greater than or equal to the first variance threshold, the linearity of the corresponding second pixel point set is determined to be unstable, such as Fig.13 As shown, the second pixel point set 102 whose variance exceeds the first threshold is identified. The distribution of these point sets is far from the straight line and may be affected by noise or outliers. These point sets are marked as unstable, which means that they may need to be further processed or eliminated in subsequent steps. When the variance of adjacent pixel points is less than or equal to the second variance threshold, the linearity of the corresponding second pixel point set is determined to be stable, such as Fig.12 As shown, second pixel point sets 102 with variances lower than a second threshold are identified, and the distribution of these point sets is dense and close to a straight line. These point sets are marked as stable, indicating that they can be used for straight line fitting without further optimization.

[0095] When the variance of adjacent pixel points is less than the first variance threshold and greater than the second variance threshold, the linearity of the corresponding second pixel point set is determined to be relatively stable, and the second pixel point set 102 with a variance between the two thresholds is identified. Although these point sets are not completely straight lines, such as Fig.11 As shown, the straight line is good and may only need slight optimization. These point sets are marked as relatively stable, meaning that they can be candidates for straight line fitting but may need further screening or optimization.

[0096] The threshold setting needs to be determined according to the actual application scenario and image features, and may need to be optimized through experiments or data analysis. Variance calculation can be based on the distance between adjacent pixels or the distance between the pixel and the fitted line. The linearity evaluation can be one-dimensional (considering only the X-axis or Y-axis direction) or two-dimensional (considering the X-axis and Y-axis directions).

[0097] Further, the linearity of the second pixel point set is optimized according to the linearity of the second pixel point set to form a third pixel point set, including: when the linearity of the second pixel point set is unstable, the scene images of the first number of frames before the current frame are determined, the first pixel point set of the scene image of the current frame is corrected according to the mean of the linear parameters of the scene images of the first number of frames, the scene image of the current frame is denoised, the region of interest in the denoised scene image is determined, and the first pixel point set is regenerated according to the deep learning network model and the region of interest; when the linearity of the second pixel point set is stable, the second pixel point set is subjected to sliding average filtering to generate the third pixel point set ; When the linearity of the second pixel point set is relatively stable, match the second pixel point set of the scene image of the current frame with the second pixel point set of the scene image of the previous frame; calculate the relative distance difference between each pixel point of the scene image of the current frame and the matched pixel point of the scene image of the previous frame, and determine the second pixel point corresponding to the unstable linearity in the scene image of the current frame according to the relative distance difference; determine two third pixel points adjacent to the second pixel point and with stable linearity, and determine the mean pixel point according to the average value of the two third pixel points; correct the second pixel point according to the mean pixel point, perform sliding average filtering on the corrected second pixel point, and generate a third pixel point set.

[0098] When optimizing linearity, targeted adjustments are mainly made based on the different linearities of the second pixel point set. Specifically, when the linearity of the second pixel point set is unstable, the scene images of the first number of frames before the current frame are determined, and the first pixel point set of the scene image of the current frame is corrected according to the mean of the straight line parameters of the scene images of the first number of frames, and the scene image of the current frame is denoised, and the first pixel point set of the scene image of the current frame is re-identified, and the current frame is corrected using the mean of the straight line parameters of the previous frames. The current frame is denoised, and the pixel point set of the current frame is re-identified and generated. The severely unstable point set is improved through historical data and denoising technology to improve the accuracy of line detection. It should be added that for the unstable second pixel point set, it is necessary to perform automatic image processing (for example, region of interest recognition) on the image of the current frame, put the denoised and intercepted image into the deep learning network model, and regenerate the first pixel point set.

[0099] When the linearity of the second pixel point set is stable, the second pixel point set is processed by sliding average filtering to generate a third pixel point set; when the linearity of the second pixel point set is relatively stable, the second pixel point set of the scene image of the current frame is matched with the second pixel point set of the scene image of the previous frame. It can be understood that the point set that has performed well is processed, and the second pixel point set is processed by sliding average filtering to further smooth the stable point set, reduce small fluctuations, and improve the accuracy of straight line fitting.

[0100] like Fig.14 and Fig.15 As shown, when the linearity of the second pixel point set is relatively stable, the point set between stable and unstable is processed, and the second pixel point set of the scene image of the current frame is matched with the second pixel point set of the scene image of the previous frame, that is, the pixel point set of the current frame is matched with the pixel point set of the previous frame; the relative distance difference between the pixel points of the scene image of each current frame and the matched pixel points of the scene image of the previous frame is calculated, and the second pixel point 1042 corresponding to the unstable linearity in the scene image of the current frame is determined according to the relative distance difference, that is, the relative distance difference between the matching points is calculated, and the unstable pixel point is determined according to the difference; two third pixel points 1044 adjacent to the second pixel point and with stable linearity are determined, and the mean pixel point 1046 is determined according to the average value of the two third pixel points 1044; the second pixel point is corrected according to the mean pixel point 1046, and the corrected second pixel point is subjected to sliding average filtering to generate as shown in Fig.15 The third pixel point set 106 is shown. The unstable points are corrected using the average value of the adjacent stable points, and the corrected point set is subjected to sliding average filtering. Through inter-frame comparison and local correction, the linearity of the relatively stable point set is improved while retaining the original stable features.

[0101] It can be understood that using data from previous frames to correct the current frame helps maintain temporal continuity and stability.

[0102] Denoising involves a variety of image processing techniques, such as median filtering and Gaussian filtering. Sliding average filtering helps smooth data and reduce random fluctuations, but attention should be paid to the selection of window size to avoid over-smoothing. Inter-frame matching involves techniques such as feature point matching or optical flow estimation. Local correction uses adjacent stable points to correct unstable points, maintaining local geometric continuity.

[0103] like Figure 5 As shown, as a preferred solution of this embodiment, the straight line parameters are combined with a single-frame scene image to generate the world coordinate information of the marker, including:

[0104] Step S1082: Acquire external parameters of the image sensor;

[0105] Step S1084: Perform inverse perspective transformation on the third pixel point set according to the external parameters of the image sensor to generate world coordinate information of the marker.

[0106] In this embodiment, when generating world coordinate information, it will be processed according to the external parameters of the image sensor. First, the external parameters of the image sensor are obtained to determine the position and posture information of the camera relative to the world coordinate system. The external parameters usually include a rotation matrix and a translation vector, which describe the relationship between the camera coordinate system and the world coordinate system. The specific acquisition method includes but is not limited to camera calibration, sensor fusion or actual estimation, wherein camera calibration is to calibrate the camera through a calibration plate or known feature points, and sensor fusion is to estimate the position and posture of the camera by combining data from other sensors (such as IMU, GPS), and real-time estimation can be understood as the need to estimate the changes in external parameters in real time in a dynamic environment.

[0107] Finally, the third pixel point set is inversely transformed according to the external parameters of the image sensor, and the pixel coordinates on the image plane are converted into three-dimensional coordinates in the world coordinate system. Specifically, the camera intrinsic parameter matrix can be used to convert the pixel coordinates into the coordinates in the camera coordinate system, or the external parameters (rotation matrix and translation vector) can be used to convert the points in the camera coordinate system into the world coordinate system.

[0108] Among them, the inverse perspective transformation is based on the pinhole camera model and uses homogeneous coordinates and matrix operations to achieve coordinate transformation.

[0109] The world coordinate information of the marker that is finally generated is the position information of the marker in the world coordinate system. For example, the navigation path 110 can be directly generated, so as to be used for tasks such as navigation, path planning, and obstacle avoidance.

[0110] like Fig.16 As shown, the system can detect straight line features in the image (such as Fig.16 The linear parameters 108 in the image (such as the edge of the road, markings, etc.) are converted into coordinate information in the real world, which is crucial for applications such as pavers that require precise positioning and navigation. This conversion enables the machine to "understand" its position in the real world and make accurate navigation decisions.

[0111] As a preferred embodiment, Figure 6 As shown, the line parameters are combined with a single-frame scene image to generate the world coordinate information of the marker, including:

[0112] Step S1092: determining scene images of a second number of frames before the current frame;

[0113] Step S1094: determining calibration information according to scene images of a second number of frames before the current frame, the calibration information being an average value of world coordinate information of the marker in scene images of a second number of frames before the current frame;

[0114] Step S1096: determining a relative distance difference according to the world coordinate information of the marker in the current frame and the calibration information;

[0115] Step S1098: Determine the world coordinate information of the marker in the current frame based on the relative distance difference.

[0116] In this embodiment, using historical frame data to calibrate and verify the world coordinate information of the current frame marker is more suitable for application scenarios that require high precision and stability, such as paver navigation. Specifically, first determine the scene image of the second number of frames before the current frame, select a certain number of historical frames as references, and use historical data to provide more stable and reliable reference information. Among them, the specific number of frames needs to balance real-time and stability, and needs to consider the speed of equipment movement and the speed of environmental changes.

[0117] Then, the calibration information is determined based on the historical frame, and the mean of the world coordinates of the markers in the historical frame is calculated as the calibration information. The world coordinates of the markers in the selected historical frame are averaged to provide a stable reference point and reduce the impact of single-frame errors.

[0118] Then calculate the relative distance difference between the current frame and the calibration information to evaluate the consistency of the current frame coordinate information with the historical data. Calculating the distance difference between the world coordinates of the marker in the current frame and the calibration information provides a mechanism for detecting outliers, which can be used to evaluate the reliability of the coordinate information of the current frame.

[0119] Finally, the world coordinate information of the marker in the current frame is determined based on the relative distance difference, and the final world coordinate information is obtained based on the historical data and the current data. If the difference is within an acceptable range, the coordinate information of the current frame can be used directly. If the difference is too large, adjustment or smoothing may be required. This can improve the accuracy and stability of the coordinate information.

[0120] The number of historical frames needs to be selected based on the application scenario and system performance to determine the appropriate number of historical frames.

[0121] When calculating the calibration information, it is possible to consider assigning different weights to different historical frames.

[0122] It should also be noted that the world coordinate systems of all frames must be consistent or have a clear conversion relationship.

[0123] As a preferred solution of this embodiment, Figure 7As shown, the line parameters are combined with a single-frame scene image to generate the world coordinate information of the marker, including:

[0124] Step S1102: when the relative distance difference is less than or equal to a preset threshold, the world coordinate information of the marker in the current frame is used as the world coordinate information of the marker;

[0125] Step S1104: when the relative distance difference is greater than a preset threshold, the first pixel point set of the scene image of the current frame is re-determined according to the average value of the straight line parameters of the scene images of the third number of frames before the current frame, the region of interest in the denoised scene image is determined, and the first pixel point set is regenerated according to the deep learning network model and the region of interest to generate new straight line parameters;

[0126] Step S1106: Combine the new straight line parameters with the single-frame scene image to generate the world coordinate information of the marker.

[0127] When the relative distance difference is less than or equal to the preset threshold, it is considered that the difference between the coordinate information of the current frame and the calibration information of the historical frame is within an acceptable range, and the world coordinate information of the marker in the current frame is directly used. This maintains the real-time performance and response speed of the system, the current frame data is reliable, and unnecessary complex calculations are avoided.

[0128] When the relative distance difference is greater than a preset threshold, the difference between the coordinate information of the current frame and the calibration information of the historical frame exceeds an acceptable range. The pixel point set of the current frame is re-determined using the historical frame data, and the current frame image is denoised to determine the region of interest in the denoised scene image. According to the deep learning network model and the region of interest, the first pixel point set is regenerated, new straight line parameters are regenerated, and the world coordinate information is recalculated using the new straight line parameters, thereby providing an error correction mechanism for abnormal situations and improving the accuracy of coordinate information by utilizing historical data and image processing technology.

[0129] The determination of the preset threshold value needs to set a suitable threshold value according to factors such as application scenario, device movement speed, environmental changes, etc. The preset threshold value may also be a dynamic threshold value, which is adaptively adjusted according to system status and environmental conditions.

[0130] The specific numbers of the first number of frames, the second number of frames, and the third number of frames may be the same or different.

[0131] Further, the first number of frames is 3-8 frames, and / or the second number of frames is 3-8 frames, and / or the third number of frames is 3-8 frames.

[0132] Further, the first number of frames is 5 frames, and / or the second number of frames is 5 frames, and / or the third number of frames is 5 frames.

[0133] like Figure 8 As shown, in another embodiment of the present application, a navigation device 900 for a road machinery is proposed, including: an image acquisition module 901, used to acquire continuous scene images through an image sensor, and the continuous scene images include multiple single-frame scene images; a marker detection module 902, used to determine a first pixel point set in a single-frame scene image according to a deep learning network model; a straight line parameter generation module 903, used to screen the first pixel point set to form a second pixel point set; perform linear optimization on the second pixel point set to form a third pixel point set; perform straight line fitting on the third pixel point set to generate straight line parameters; a coordinate generation module 904, used to combine the straight line parameters with the single-frame scene image to generate world coordinate information of the marker; a control module 905, used to generate a navigation path according to the world coordinate information in multiple single-frame scene images, and control the road machinery to travel along the navigation path.

[0134] In this embodiment, the accuracy and stability of machine vision in complex environments can be improved, especially in the application of automated construction equipment such as pavers. Specifically, by acquiring continuous scene images, ensuring that there is sufficient data for dynamic analysis and processing, providing a continuous image data stream, and providing a basis for stable detection in dynamic environments.

[0135] On this basis, by detecting the markers in a single-frame scene image, identifying and locating the markers in each frame image, such as curbs or white lime lines, a first pixel point set corresponding to the markers is generated, and the first pixel point set is fitted. Specifically, a mathematical fitting method (such as the least squares method) can be used to fit a straight line to the detected pixel point set.

[0136] The line parameters are extracted from the pixel point set, and the position and direction of the marker in the image can be determined based on the line parameters. Finally, the line parameters are combined with the single-frame scene image to generate the world coordinate information of the marker. It can be understood that the generation of a single-frame scene image can be determined based on the internal and external parameters of the camera or other image acquisition device, and the two-dimensional image coordinates can be converted into three-dimensional world coordinates, thereby providing the specific position of the marker in the real world.

[0137] Among them, the acquisition of continuous images ensures that the system can adapt to dynamically changing environments, such as the movement of the paver under different road conditions. The detection of single-frame images allows the system to capture the state of the marker at every moment. The fitting of the pixel point set is a key step that directly affects the accuracy of the line parameters and thus the final world coordinate information. The generation of world coordinates as the ultimate goal enables the paver to accurately know its position relative to the road marker, thereby achieving precise control.

[0138] Optionally, the marker detection module 902 is also used to input a single-frame scene image into a deep learning network model; through the deep learning network model, identify the marker in the single-frame scene image to obtain a first pixel point set corresponding to the marker.

[0139] The line parameter generation module 903 is further used to determine a screening threshold of the first pixel point set; and screen the pixel points in the Y-axis direction in the single-frame scene image according to the screening threshold to form a second pixel point set.

[0140] The straight line parameter generation module 903 is also used to determine the world coordinates of a reference object on the road surface machinery, where the relative position of the reference object and the image sensor is fixed; convert the world coordinates of the reference object into the image coordinates of the reference object according to the external parameters of the image sensor; and determine the first pixel point set in the area of ​​interest as the second pixel point set according to the image coordinates of the reference object and the image coordinates of the marker.

[0141] The straight line parameter generation module 903 is also used to determine the first pixel point in the second pixel point set in the X-axis direction in the single-frame scene image; perform variance statistics on adjacent first pixel points to obtain adjacent pixel point variances; determine the linearity of the second pixel point set based on the adjacent pixel point variances; and optimize the linearity of the second pixel point set based on the linearity of the second pixel point set to form a third pixel point set.

[0142] The straight line parameter generation module 903 is also used to set a first variance threshold and a second variance threshold, the first variance threshold is greater than the second variance threshold; when the variance of adjacent pixel points is greater than or equal to the first variance threshold, the linearity of the corresponding second pixel point set is determined to be unstable; when the variance of adjacent pixel points is less than or equal to the second variance threshold, the linearity of the corresponding second pixel point set is determined to be stable; when the variance of adjacent pixel points is less than the first variance threshold and greater than the second variance threshold, the linearity of the corresponding second pixel point set is determined to be relatively stable.

[0143] The straight line parameter generation module 903 is also used to, when the linearity of the second pixel point set is unstable, determine the scene images of the first number of frames before the current frame, correct the first pixel point set of the scene image of the current frame according to the mean of the straight line parameters of the scene images of the first number of frames, denoise the scene image of the current frame, determine the region of interest in the denoised scene image, regenerate the first pixel point set according to the deep learning network model and the region of interest, and re-fit the first pixel point set of the scene image of the current frame; when the linearity of the second pixel point set is stable, perform sliding average filtering on the second pixel point set to generate a third pixel point set; when the linearity of the second pixel point set is stable, perform sliding average filtering on the second pixel point set to generate a third pixel point set; When the linearity of two pixel point sets is relatively stable, the second pixel point set of the scene image of the current frame is matched with the second pixel point set of the scene image of the previous frame; the relative distance difference between each pixel point of the scene image of the current frame and the matched pixel point of the scene image of the previous frame is calculated, and the second pixel point corresponding to the unstable linearity in the scene image of the current frame is determined according to the relative distance difference; two third pixel points adjacent to the second pixel point and with stable linearity are determined, and the mean pixel point is determined according to the average value of the two third pixel points; the second pixel point is corrected according to the mean pixel point, and the corrected second pixel point is subjected to sliding average filtering to generate a third pixel point set.

[0144] The coordinate generation module 904 is also used to obtain the external parameters of the image sensor; perform an inverse perspective transformation on the third pixel point set according to the external parameters of the image sensor to generate world coordinate information of the marker; determine the scene image of the second number of frames before the current frame; determine the calibration information according to the scene image of the second number of frames before the current frame, the calibration information being the average of the world coordinate information of the marker in the scene image of the second number of frames before the current frame; determine the relative distance difference according to the world coordinate information of the marker in the current frame and the calibration information; when the relative distance difference is less than or equal to a preset threshold, use the world coordinate information of the marker in the current frame as the world coordinate information of the marker; when the relative distance difference is greater than the preset threshold, re-determine the first pixel point set of the scene image of the current frame according to the average value of the straight line parameters of the scene image of the third number of frames before the current frame, denoise the scene image of the current frame, and re-fit the first pixel point set of the scene image of the current frame to generate new straight line parameters; combine the new straight line parameters with the single-frame scene image to generate the world coordinate information of the marker.

[0145] like Fig. 9As shown, an embodiment of the present application further provides an electronic device 100, including a processor 1110, a memory 1109, and a program or instruction stored in the memory 1109 and executable on the processor. When the program or instruction is executed by the processor 1110, each process of the embodiment of the navigation method of the above-mentioned road machinery is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0146] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned electronic devices and non-electronic devices.

[0147] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned navigation method embodiment of the road machinery are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0148] The embodiment of the present application also provides a computer program product for a computer, including a software code portion. When the product is run on a computer, the software code portion is used to execute and implement the various processes of the above-mentioned navigation method embodiment of the road machinery, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0149] like Fig.17 As shown, an embodiment of the present application further provides a road machinery 200, including a body structure 201 and a navigation device 900 of the road machinery, wherein an image sensor 202 and a reference object 204 are provided on the body structure 201, the image sensor 202 is used to acquire scene images, and the reference object 204 is used to automatically capture an area of ​​interest.

[0150] The embodiment of the present application also provides a filtering enhancement method and system corresponding to the embodiment of the linear detection of the image of the paver. The system logic can solve the problem that the deep learning model fails to detect some scenes and is in a jumping state, and provides stable positioning data, so that the paver can walk normally and automatically, and stop after failing to accurately detect and locate for a long time. The scheme proposes a single-frame image jump point discrimination algorithm to determine whether the single-frame detection point is stable and meets the linearity of the detection target. A joint judgment algorithm for the front and back frames is proposed. By comparing the point positions of the front and back detection frames, the jump points of the current frame are eliminated using the Gaussian distribution, and the stable points of the current frame are used for linear interpolation. An automatic distance calibration algorithm is proposed. When the paver is driving automatically, the result of the previous stable frame is used to determine whether the detection distance of the current frame is normal. An automatic ROI algorithm is proposed. When the paver is driving automatically, the straight line parameters of the previous frame are used to find the approximate area of ​​the current frame in real time, so as to automatically intercept the target area for secondary reasoning after the current frame detection fails, thereby improving the stability of the model detection.

[0151] Specifically, Fig.10 As shown, the filtering enhancement method is as follows:

[0152] Step S202, image preprocessing (distortion correction, etc.). Use an image sensor to acquire a scene image, calibrate the sensor to obtain internal and external parameters. Use the internal parameters to perform preprocessing such as distortion correction on the sensor.

[0153] Step S204: deep learning model detection: the pre-processed image is put into the deep learning network model to obtain a number of target detection points.

[0154] Step S206, extract the region of interest from the detection points of the single frame image and perform linear judgment. Intercept the region of interest for the obtained detection points. The region of interest can be determined by manually setting the threshold in the Y direction of the image, or by setting the position of the guide rod of the paver, automatically calculating the position of the guide rod in the image through the calibrated camera external parameters, and automatically delineating the region of interest using the intersection of the detection line and the guide rod.

[0155] Furthermore, for the target detection points of interest, a linearity judgment is performed. Since the detection target direction is consistent with the driving direction of the paver, the pixels of the detection points on the X-axis can be used for judgment. When the detection points are distributed on a straight line, the X-axis direction is evenly distributed, and the pixel differences between the points are in an arithmetic progression. The variance statistics can be performed by the difference between adjacent points and separated points in the X-axis direction to determine whether the point is a straight line. Two variance thresholds are set. When the variance of the current frame is greater than the high threshold, it is considered that the detection jump of the frame is large and the detection fails; when the variance is less than the low threshold, the detection of the frame is considered stable; when the variance is greater than the low threshold and less than the high threshold, the target detection of the frame is determined to be a straight line, but there are individual jump points in the detection points.

[0156] Step S208, perform stability judgment on the corresponding detection points of the previous and next frames, and remove the jump points. If the current frame is judged to be a straight line, but there are jump points, compare the corresponding points of the previous and next frames and remove the jump points. Taking advantage of the slow travel speed of the paver, calculate the corresponding point difference values ​​of the series of points detected in the previous and next frames on the pixels in the X-axis direction, and count their corresponding positions. Calculate the mean and variance of the difference. And perform Gaussian distribution on it. Find the corresponding pixel points of the previous and next frames that jump outside the threshold, and remove the jump points. Then use the stable points of the current frame to insert new detection points at the removal point position using the mean of two adjacent points. Maintain the stability of the current frame.

[0157] Step S210, performing straight line fitting on the acquired detection points to obtain straight line parameters. Perform straight line fitting on the detection points of the filtered stable frame to obtain straight line parameters as parameters for subsequent automatic ROI.

[0158] Step S212: Perform perspective transformation on the filtered points to obtain the coordinates of the points in the world coordinate system. Perform inverse perspective transformation using the external parameters of camera calibration to obtain the world coordinates of the target detection points.

[0159] Step S214, linear fitting is performed on the world coordinate point to obtain the horizontal coordinate of the designated position. Linear fitting is performed to obtain the horizontal distance of the roadside, lime line and other markers at the designated position, and the angle between the marker and the paver.

[0160] Step S216, the horizontal coordinate is calibrated and the obtained distance is calibrated.

[0161] Step S218, determine whether it is stable, specifically determine whether the detection result of the frame is stable. If it is stable, then step S2182, generate a stable result; otherwise, step S2184, generate an unstable result. When the paver starts the automatic walking command, a container is created to save the distance result of the stable frame. The container size is set to 5 frames. If there is new data, the oldest data will be automatically removed. The container size can be modified manually. The distance in the container is used to calculate its mean, and the mean is set to the calibration value, which is updated in real time with the stable data. Then compare the distance result of the current frame with the calibration value. If it exceeds the set threshold, it is determined to be a detection failure.

[0162] Since the paver is traveling at a slow speed, a scene that cannot be recognized appears for a long time and is always in a state of detection failure. Therefore, for the unstable results generated in step S206 and step S218, that is, the image of the failed frame, secondary reasoning is performed through step S224. Using the detection line parameters saved in the stable frame in step S210, in step S226, the image is automatically ROI, that is, the ROI area is automatically generated around the marking objects such as the curb or lime line.

[0163] Step S228, intercept the area of ​​interest near the target line, and blacken the remaining interference area. That is, blacken the background outside the area to remove the influence of surrounding interference on the detection model, re-detect and filter the points, and repeat steps S204 to S218. If the result after the secondary reasoning meets the requirements, replace the result of the primary reasoning. If the result of the secondary reasoning is not within the threshold range, it is determined that the detection has failed.

[0164] Step S220, mean filtering is performed on the points that meet the requirements, that is, the results that meet the threshold requirements are processed by sliding mean to reduce the fluctuation of the output value. Step S222, the sliding window calculates the calibration value, and the result is input to the controller, so as to update the calibration value in real time, and continue to perform step S216 to determine the calibration value of its horizontal coordinate.

[0165] Step S230: Output to control: Send the lateral distance, angle, status position and other information of the detected marker to the control module.

[0166] Based on the pixel difference statistics of adjacent points and separated points in a single frame, the waveform linear points that fail to be detected can be filtered out.

[0167] Based on the pixel difference of corresponding points in the previous and next frames, Gaussian statistical distribution can be performed to eliminate single-point pixel jumps in the case of good linearity.

[0168] In the automatic state, the calibration value is judged for each frame result, and the driving characteristics of the paver are used to eliminate abnormal frame data.

[0169] For detection failure frames due to interference from the surrounding environment, the ROI area can be automatically drawn to remove surrounding interference items, and re-inference can be performed to obtain stable results.

[0170] Based on the above specific embodiments, first, an image sensor is used to acquire a scene image, and the sensor is calibrated to obtain internal and external parameters. Distortion correction and other preprocessing are performed using the internal parameters. The preprocessed image is input into a deep learning network model to obtain a number of target detection points.

[0171] The region of interest (ROI) is extracted for the detected points. The ROI can be automatically delineated by manually setting the threshold in the Y direction or automatically calculating the position of the paver guide rod in the image through the camera external parameters and using the intersection of the detection line and the guide rod.

[0172] The linearity of the target detection points within the ROI is judged. Since the detection target direction is consistent with the driving direction of the paver, the pixels in the X-axis direction should be evenly distributed. The variance statistics are performed through the difference between adjacent points and separated points to determine whether the detection point is on a straight line, and the stability of the detection is determined based on the variance threshold.

[0173] In the case of jump points, the jump points are removed by comparing the positions of the detection points of the previous and next frames. The pixel difference between the previous and next frames in the X-axis direction is used for Gaussian distribution statistics to find the jump points and insert new detection points with the mean of adjacent points to maintain the stability of detection.

[0174] Perform straight line fitting on the stable detection points after filtering to obtain the straight line parameters, which are used as the parameters of the subsequent automatic ROI.

[0175] The inverse perspective transformation is performed using the external parameters of camera calibration to obtain the world coordinates of the target detection point, and a straight line fitting is performed to obtain the lateral distance of the marker at the specified position and its angle with the driving direction of the paver.

[0176] Create a container to save the distance results of the stable frame, calculate its mean as the calibration value, and update it in real time. Compare the distance result of the current frame with the calibration value, and if it exceeds the set threshold, it is judged as a detection failure.

[0177] For frames that fail to be detected, the detection line parameters of the stable frame are used to automatically generate the ROI area, remove interference items, and re-detect and filter. If the secondary reasoning result meets the requirements, the primary reasoning result is replaced, otherwise it is judged as a detection failure.

[0178] The results that meet the threshold requirements are processed by sliding average to reduce output value fluctuations and update the calibration value at the same time.

[0179] The lateral distance, angle, status position and other information of the detected marker are sent to the control module.

[0180] This method enhances the stability of detection by filtering and calibrating the detection points of a single frame and previous and next frames, ensuring that the paver can obtain accurate positioning data during automatic driving. In particular, the generalization ability and stability of the detection model are significantly improved by using automatic ROI and secondary reasoning technology in the case of environmental interference and detection failure.

[0181] In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0182] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or module referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, should not be understood as a limitation on the present invention.

[0183] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0184] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A navigation method for a road machine, It is characterized in that For a road machine having an image sensor, the navigation method of the road machine comprises: Acquiring continuous scene images through an image sensor, wherein the continuous scene images include a plurality of single-frame scene images; Determine a first pixel point set of the marker in the single-frame scene image according to the deep learning network model; Screening the first pixel point set to form a second pixel point set; performing linear optimization on the second pixel point set to form a third pixel point set; Performing straight line fitting on the third pixel point set to generate straight line parameters; Combining the straight line parameters with the single-frame scene image to generate world coordinate information of the marker; A navigation path is generated according to the world coordinate information in the plurality of single-frame scene images, and the road machine is controlled to travel along the navigation path.

2. The navigation method for a road machine according to claim 1, It is characterized in that The step of determining a first pixel point set of the marker in the single-frame scene image according to the deep learning network model includes: Inputting the single-frame scene image into a deep learning network model; The deep learning network model is used to identify the marker in the single-frame scene image and obtain a first pixel point set corresponding to the marker.

3. The navigation method for a road machine according to claim 1, It is characterized in that The screening of the first pixel point set to form a second pixel point set includes: Determining a screening threshold of the first pixel point set; According to the screening threshold, the pixel points in the Y-axis direction in the single-frame scene image are screened to form a second pixel point set.

4. The navigation method for a road machine according to claim 1, It is characterized in that The screening of the first pixel point set to form a second pixel point set includes: Determining the world coordinates of a reference object on the road surface machine, wherein the relative position of the reference object and the image sensor is fixed; Converting the world coordinates of the reference object into the image coordinates of the reference object according to the external parameters of the image sensor; According to the image coordinates of the reference object and the image coordinates of the marker, a first pixel point set in the region of interest is determined as the second pixel point set.

5. The navigation method for a road machine according to claim 1, It is characterized in that The linear optimization of the second pixel point set to form a third pixel point set includes: Determine a first pixel point of the second pixel point set in the X-axis direction in the single-frame scene image; Performing variance statistics on adjacent first pixel points to obtain variances of adjacent pixel points; Determining the linearity of the second pixel point set according to the variance of the adjacent pixel points; The linearity of the second pixel point set is optimized according to the linearity of the second pixel point set to form a third pixel point set.

6. The navigation method for a road machine according to claim 5, It is characterized in that The determining the linearity of the second pixel point set according to the variance of the adjacent pixel points includes: Setting a first variance threshold and a second variance threshold, wherein the first variance threshold is greater than the second variance threshold; When the variance of the adjacent pixel points is greater than or equal to a first variance threshold, the linearity of the corresponding second pixel point set is determined to be unstable; When the variance of the adjacent pixel points is less than or equal to a second variance threshold, the linearity of the corresponding second pixel point set is determined to be stable; When the variance of the adjacent pixel points is less than the first variance threshold and greater than the second variance threshold, the linearity of the corresponding second pixel point set is determined to be relatively stable.

7. The navigation method for a road machine according to claim 5, It is characterized in that The step of optimizing the linearity of the second pixel point set according to the linearity of the second pixel point set to form a third pixel point set includes: When the linearity of the second pixel point set is unstable, determining scene images of a first number of frames before the current frame, correcting the first pixel point set of the scene image of the current frame according to the mean of the straight line parameters of the scene images of the first number of frames, performing denoising on the scene image of the current frame, determining a region of interest in the denoised scene image, regenerating the first pixel point set according to the deep learning network model and the region of interest, and refitting the first pixel point set of the scene image of the current frame; When the linearity of the second pixel point set is stable, performing a sliding average filtering process on the second pixel point set to generate a third pixel point set; When the linearity of the second pixel point set is relatively stable, the second pixel point set of the scene image of the current frame is matched with the second pixel point set of the scene image of the previous frame; the relative distance difference between each pixel point of the scene image of the current frame and the matched pixel point of the scene image of the previous frame is calculated, and the second pixel point corresponding to the unstable linearity in the scene image of the current frame is determined according to the relative distance difference; two third pixel points adjacent to the second pixel point and whose linearity is stable are determined, and the mean pixel point is determined according to the average value of the two third pixel points; the second pixel point is corrected according to the mean pixel point, and the corrected second pixel point is subjected to sliding average filtering to generate a third pixel point set.

8. The navigation method for a road machine according to claim 1, It is characterized in that The step of combining the straight line parameters with the single-frame scene image to generate the world coordinate information of the marker includes: Acquiring external parameters of the image sensor; Performing an inverse perspective transformation on the third pixel point set according to the external parameters of the image sensor to generate world coordinate information of the marker; Determine scene images of a second number of frames before the current frame; Determine calibration information according to the scene images of the second number of frames before the current frame, wherein the calibration information is an average value of the world coordinate information of the marker in the scene images of the second number of frames before the current frame; Determine a relative distance difference according to the world coordinate information of the marker in the current frame and the calibration information; When the relative distance difference is less than or equal to a preset threshold, the world coordinate information of the marker in the current frame is used as the world coordinate information of the marker; When the relative distance difference is greater than a preset threshold, re-determine the first pixel point set of the scene image of the current frame according to the average value of the straight line parameters of the scene images of the third number of frames before the current frame, denoise the scene image of the current frame, determine the region of interest in the denoised scene image, regenerate the first pixel point set according to the deep learning network model and the region of interest, and generate new straight line parameters; The new straight line parameters are combined with the single-frame scene image to generate world coordinate information of the marker.

9. A navigation device for a road machine, It is characterized in that include: An image acquisition module, used for acquiring continuous scene images through an image sensor, wherein the continuous scene images include a plurality of single-frame scene images; A marker detection module, used to determine a first pixel point set of the marker in the single-frame scene image according to a deep learning network model; A line parameter generation module, used for screening the first pixel point set to form a second pixel point set; Performing linear optimization on the second pixel point set to form a third pixel point set; performing straight line fitting on the third pixel point set to generate straight line parameters; A coordinate generation module, used for combining the straight line parameters with the single-frame scene image to generate world coordinate information of the marker; A control module is used to generate a navigation path according to the world coordinate information in the plurality of single-frame scene images, and control the road machine to travel along the navigation path.

10. A road machine, It is characterized in that include: A fuselage structure, on which an image sensor and a reference object are disposed; The navigation device for a road machine as described in claim 9 is arranged in the fuselage structure.

Images