Method that analyses the textured structure of the road

Abstract

The present invention is a method for estimating variable road surface defects by using images taken from the road in a vehicle having at least one camera (20) which acts as at least one image sensor, including at least one internal sensor (10) and / or virtual sensor obtained from other sensors on the vehicle to calculate the road elevation and profile in the X-Y-Z directions from the road to the vehicle, characterized by the following; in order to work under the supervision of at least one controller on the vehicle or in the cloud to evaluate the image from said camera (20): determining measurement points by taking the image from the camera (20) at predetermined fixed frequencies; measuring the pattern / color / texture structure over the image taken from the camera (20) during estimation and reliability scoring at t=0 in two different categories; in order to be used in learning by classifying the repetitive and less reliable pattern / color / texture structure; using the error between the prediction at t<0 and the measurement at t=0 for weighting during the estimation; calibrating the highly reliable pattern / color / texture structure with physical measurement results and using it in weighting during forecasting.

Description

[0001] METHOD THAT ANALYSES THE TEXTURED STRUCTURE OF THE ROAD

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to a method for estimating variable road surface defects by using images taken from the road in a vehicle having at least one camera (20) which acts as at least one image sensor, including at least one internal sensor and / or virtual sensor obtained from other sensors on the vehicle to calculate the road elevation and profile in the X-Y-Z directions from the road to the vehicle.

[0004] In particular, the present invention is a prediction structure made with deep learning, machine learning, etc. obtained from the processed data, especially in vehicles, having an image sensor and at least one physical sensor on the vehicle (and / or physical sensors on the vehicle).

[0005] PRIOR ART

[0006] Cameras have been widely used with the developing technology in the automotive sector. Thus, fully autonomous or semi-autonomous driving has been developed and the vehicle can be safely driven independent of the driver. Analysis is realized by processing the environmental data received from the camera, and the driver can be warned with the road condition and / or the vehicle can be driven by itself.

[0007] Visual Sensing systems, such as cameras, can offer centimeter scale accuracy to detect the roughness and undulation of the road surface. Accuracy varies depending on factors such as lighting, road surface texture and reflection. It can detect the road surface with the disparity image taken by a stereo camera and depth estimation. The monocular camera is also used to create a disparity image with sequentially captured images. In these methods, different image processing and object recognition techniques are used to identify objects with the help of shape and color objects. At the same time, these methods comprise an internal algorithm that evaluates the measurement security of the pixel group. However, these methods are not based on physical facts and only compare a group of pixels. For this reason, it cannot produce a durable solution against the changing texture of the road surface. In traffic, the texture also changes depending on the change from road to road. The algorithms presented within these methods are calibrated before being presented, but they cannot perform continuous learning or selfevaluation based on physical facts.

[0008] Prior art methods only estimate visual perception in a limited way. It does not have continuous learning and automatic calibration capability. On the other hand, tactile road profile perception can provide insight into road characteristics, but the vehicle is not capable of foresight. Therefore, visual perception alone is not sufficient for selfassessment measurement security.

[0009] In the TURK PATENT document with publication number 2021 / 009190, the method that enables the estimation of the road profile is mentioned. In this solution, changes in the road surface are not followed, but major form changes on the road are followed.

[0010] As a result, all abovementioned problems have made it necessary to make an improvement in the relevant technical field.

[0011] OBJECT OF THE INVENTION

[0012] The present invention aims to eliminate the abovementioned problems and to make a development in the relevant technical field.

[0013] The main object of the present invention is to reveal the system and method structure that provides the user with a more comfortable driving and provides the necessary information by learning the pattern and texture of the road in vehicles.

[0014] Another object of the present invention is to ensure that centimetre scale surface defects (roughness and wavy structure) on the road are accurately detected by means of the camera on the vehicle. With this information, active or semi-active suspension systems can be triggered, thus increasing driver comfort.

[0015] Another object of the present invention is to increase reliability by performing virtual measurement for visual measurement.

[0016] Another object of the present invention is to introduce a method that can calibrate virtual measurement by learning the pattern and texture of the road. Another object of the present invention is to provide the ability to combine tactile and visual information on the vehicle.

[0017] Another object of the present invention is to enable visual prediction and continuous learning about the texture, patterns and roughness of objects and surfaces on the image, by using machine learning and neural networks.

[0018] BRIEF DESCRIPTION OF THE INVENTION

[0019] The present invention relates to method that analyses the textured structure of the road so as to fulfil all aims mentioned above and will be obtained from the following detailed description.

[0020] Within the scope of the invention, the vehicle in which the method operates has an image sensor and at least one physical sensor on the vehicle (and / or physical sensors on the vehicle), and the method is an estimation structure made with deep learning, machine learning, etc. obtained from the processed data.

[0021] The present invention is a method for estimating variable road surface defects by using images taken from the road in a vehicle having at least one camera which acts as at least one image sensor, including at least one internal sensor and / or virtual sensor obtained from other sensors on the vehicle to calculate the road elevation and profile in the X-Y-Z directions from the road to the vehicle, characterized by the following; in order to work under the supervision of at least one controller on the vehicle or in the cloud to evaluate the image from said camera: determining measurement points by taking the image from the camera at predetermined fixed frequencies; ignoring maneuvering data such as traction force, braking forces and skidding forces in turns and performing learning by verifying the prediction made in a previous period (t<0) with the data received at the moment when the reference point of the vehicle reaches the relevant prediction area (t=0), measuring the pattern / color / texture structure on the image taken from the camera during prediction and scoring reliability in two different categories at t=0; in order to be used in learning by classifying the repetitive and less reliable pattern / color / texture structure; using the error between the prediction made at time t<0 and the measurement made at time t=0 in weighting during prediction; calibrating the highly reliable pattern / color / texture structure with physical measurement results and using it in weighting during prediction. In a further preferred embodiment of the invention, to enable image processing under the control of a controller, characterized by the following; a. Dividing the pixels in each image frame into equal squares; b. Calculating height values from the image frame for each frame divided for t<0 c. Matching physical measurements for the same frame for t=0; d. Finding differences / gaps for t<0 and t=0 in the same divided frame e. For pixel groups consisting of divided squares, finding groups containing anomalies in the same region; f. Comparing groups containing anomalies with the remaining pixels in the image frame for pattern / color / texture differences.

[0022] In another preferred embodiment of the invention, the pixels in each image frame are processed by dividing them in matrix form.

[0023] In another preferred embodiment of the invention, the center of gravity is taken as the reference point and it is the center of gravity of the virtual axes during physical measurement.

[0024] In another preferred embodiment of the invention, virtual measurement is carried out using an artificial neural network under controller control.

[0025] In another preferred embodiment of the invention, under the supervision of the controller, the measured and finalized end points are marked with coordinate-based positioning / alignment data and stored on at least one cloud server. In this context, additional parameters are taken into account when localizing the vehicle and the measurement by evaluating the surrounding factors such as buildings, objects, traffic signs, etc. from the camera.

[0026] The protection scope of the invention is specified in the claims and cannot be limited to the description made for illustrative purposes in this brief and detailed description. It is clear that a person skilled in the art can present similar embodiments in the light of the above descriptions without departing from the main theme of the invention.

[0027] BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 shows the graph showing the operation of the system. Figure 2 shows the flow diagram explaining the operation of the method subject to the invention.

[0029] Figure 3 shows a representative drawing and flow diagram explaining the process of dividing the determined road into equal dimensions.

[0030] The drawings are not intended to limit the scope of protection defined in the claims and should not be referred to in isolation without reference to the technical description in the description of the present invention for the purpose of interpreting the scope defined in those claims. The drawings in question are intended to define the invention with clarity.

[0031] DESCRIPTION OF THE REFERENCES IN FIGURES

[0032] 10. Sensor

[0033] 20. Camera

[0034] DETAILED DESCRIPTION OF THE INVENTION

[0035] In this detailed description, the inventive method that analyses the textured structure of the road is described by means of examples only for clarifying the subject matter such that no limiting effect is created.

[0036] The subject of the present invention relates to a method that enables the estimation of irregularities on variable road surfaces by using images taken from the road in vehicles.

[0037] The present invention is a method for estimating the variable road surface by using images taken from the road in a vehicle with at least one internal sensor (10) to calculate the road elevation and profile in the X-Y-Z directions coming from the road to the vehicle and with a camera (20) that acts as at least one image sensor; in order to operate under the control of at least one controller on the vehicle or on the cloud to evaluate the image taken from the said camera (20), characterized by the following;

[0038] Determining the measurement points by taking the image from the camera (20) at predetermined fixed frequencies; ignoring the maneuver data, which are the traction force, braking forces and skidding forces in turns, and learning is performed by verifying the prediction made at a previous time (t<0) with the data taken at the moment when the reference point of the vehicle reaches the relevant prediction area (t=0), measuring the pattern / color / texture structure over the image taken from the camera (20) during estimation and reliability scoring at t=0 in two different categories; using the error between the prediction made at time t<0 and the measurement made at time t=0 for weighting during prediction for use in learning by classifying the pattern / color / texture structure, which is repetitive and less reliable; calibrating the highly reliable pattern / color / texture structure with physical measurement results and using it in weighting during forecasting.

[0039] The invention can detect the pattern / color / texture structure on the road through the image frames taken by the camera (20). In this way, it is ensured that surface defects (roughness and wavy structure) in the centimeter / millimeter range on the road are accurately detected by the camera (20) on the vehicle.

[0040] Virtual estimation is achieved through the controller of the invention. If a vehicle passes the location where the estimation takes place, physical measurement is made and the two data are compared with each other. The sensors (10) of the vehicle can be used for comparison.

[0041] In a preferred embodiment of the invention, the center of gravity of the vehicle is selected as the reference point, and based on this reference point, measurements can be made with the sensor (10) for the roughness and wavy structure on the road. Within the scope of virtual estimation, the distance from the center of gravity of the vehicle to the analysed region in the image frame is calculated from the image.

[0042] In Figure 3, a drawing describing the process of dividing the road in front of the vehicle into squares of equal size is given. The following are provided; processing (via machine learning / deep learning / AI) of image sensor data, preferably camera (20) and physical sensor data (10) measured directly on the vehicle and / or physical sensor data (10) measured on the vehicle, and comparing the image-derived measurement regions corresponding to the same frames as the data from a virtual road profile measurement sensor and establishing a link between the image and the tactile measurement by comparing the anomalies in the image with the data obtained from the other method. Thanks to this link, the measurement reliability of the image is increased. Continuous learning is processed with higher reliability if a similar road texture structure measured before moment "t" is detected in the next measurement.

[0043] In another preferred embodiment of the invention, images can be taken from the camera (20) at 25Hz and lower frequencies. In this way, measurement points can be created on the road profile. This frequency may vary in alternative applications.

[0044] Within the scope of the invention, learning is carried out by verifying the data received at the moment when the reference area of the vehicle reaches the relevant prediction area (t = 0).

[0045] During the comparison, the pixels in each image frame are divided into equal squares. For each frame divided for t<0 within the created segments, the height values are calculated from the picture frame. For t=0, physical measurements are matched with measurement results for the same frame. In the next stage, the process of finding the differences / gaps for t<0 and t=0 in the same divided square is carried out. For pixel groups consisting of split frames, verification is performed by finding groups containing anomalies in the same region and comparing the groups containing anomalies with the rest of the pixels in the image frame for pattern / color / texture differences. In this way, the estimation made with the artificial neural network is tested for each data.

[0046] Within the scope of the invention, each segment contains its own profile height data. In this way, physical measurement and virtual / predictive (Artificial neural network) measurement can be compared one to one.

[0047] Within the scope of the invention, scoring of real values is provided by estimation at time t = 0. With this scoring process, the reliability level is divided into two categories. With image processing, a statistical confidence interval is calculated based on the values previously taught, the distribution of the measurement points themselves and, preferably, a few other parameters such as light intensity, etc. Within the scope of the invention, in addition to the distribution of the points with this technique, the measurement of the same points with another method is scored. In the preferred embodiment of the invention, the reliability level can be determined within a specified range, for example between 0-100. In this context, it is anticipated that the information coming from the road profile and the camera (20) has a confidence interval between 0- 100. In results with a high level of reliability, physical measurement values (measurement made with sensors (10) at time t = 0) are used as the weight coefficient in the prediction for the next moment t < 0.

[0048] In case of results with low reliability level, these repetitions are learned and if they repeat continuously, the road characteristic is classified. The detected error is used as the weight coefficient in the prediction for the next t<0.

[0049] If there is no consistent repetition in the results with low reliability level, the road characteristics are separated independently. In this way, outliers are minimized within the learning algorithm.

[0050] Thanks to the method within the scope of the invention, a more stable road profile is created.

[0051] A picture is taken over the road by means of the camera (20), which is the sensor (10) located on the glass of the vehicle with the inventive method. In this context, the data received are processed by optimizing the inventive method.

[0052] In the preferred embodiment of the invention, determining the position of the camera (20) according to the Earth coordinate is an important criterion in the process of transferring the frames taken from the camera (20) from the distance domain to the time basis.

[0053] In a preferred embodiment of the invention, it is a method for detecting and processing surface shapes (in x-y-z axes) with road (R) profiles detected by a camera (20) that acts as at least one image sensor in vehicles, characterized by carrying out the following process steps; controlling whether the images taken by said camera (20) contain height and distance (x-y axis) data or not, creating a point cloud for points, containing data points with height and distance, adding the data of the same surface shape to the point cloud, aligning points in the point cloud, clustering of aligned points and determining adjacent radius in the connection, filtering out the noise outside the cluster, converting data from space plane (distance, height, width) domain to time domain. In this way, the profile of the road is calculated. In addition to this process, a stable road profile estimation can be achieved by analysing the road surface and learning the irregularities on the road profile surface.

[0054] In a preferred embodiment of the invention, the sensor (10) enables the measurement of position changes occurring in the X-Y-Z axes at the center of gravity. In this context, there may be a sensor (10) positioned in the wheel connection areas, measuring the displacement, or a gyroscope. In another preferred embodiment of the invention, previously learned texturized structures and measurement values are tagged / marked with location data and stored in a cloud for later use. When a vehicle equipped with this system passes through the same location or similar textured structure, the data captured from the cloud can be reused and a re-evaluation of the road can be made. In another preferred embodiment of the invention, if there is a change in the path, this information can be shared. Road maintenance, whether it has been done, any deterioration in the road structure, etc.

[0055] The protection scope of the invention is specified in the appended claims and cannot be limited to the description made for illustrative purposes in this detailed description. Likewise, it is clear that a person skilled in the art can present similar embodiments in the light of the above descriptions and technical drawings without departing from the main theme of the invention.

Claims

CLAIMS1. Method for estimating variable road surface defects by using images taken from the road in a vehicle having at least one camera (20) which acts as at least one image sensor, including at least one internal sensor (10) and / or virtual sensor obtained from other sensors on the vehicle to calculate the road elevation and profile in the X-Y-Z directions from the road to the vehicle, in order to work under the supervision of at least one controller on the vehicle or on the cloud to evaluate the image received from the said camera (20): characterized by the following; Determining the measurement points by taking the image from the camera (20) at predetermined fixed frequencies; ignoring the maneuver data, which are the traction force, braking forces and skidding forces in turns, and learning is performed by verifying the prediction made at a previous time (t<0) with the data taken at the moment when the reference point of the vehicle reaches the relevant prediction area (t=0), measuring the pattern / color / texture structure over the image taken from the camera (20) during estimation and reliability scoring at t=0 in two different categories; using the error between the prediction made at time t<0 and the measurement made at time t=0 for weighting during prediction for use in learning by classifying the pattern / color / texture structure, which is repetitive and less reliable; calibrating the highly reliable pattern / color / texture structure with physical measurement results and using it in weighting during forecasting.

2. A method that enables the detection of the textured structure of the road with the image taken from the camera (20) in vehicles according to claim 1 , in order to ensure that the image is processed under the supervision of the controller, characterized by the following; a. Dividing the pixels in each image frame into equal squares; b. Calculating height values from the image frame for each frame divided for t<0 c. Matching physical measurements for the same frame for t=0; d. Finding differences / gaps for t<0 and t=0 in the same divided frame ioe. For pixel groups consisting of divided squares, finding groups containing anomalies in the same region; f. Comparing groups containing anomalies with the remaining pixels in the image frame for pattern / color / texture differences.

3. Method according to claim 2, characterized in that; the pixels in each image frame are divided and processed in matrix form to detect the textured structure of the road with the image taken from the camera (20).

4. Method that enables the textured structure of the road to be determined with the image taken from the camera (20) in vehicles according to claim 1 , characterized in that; the center of gravity is taken as the reference point and the virtual axes are the center of gravity during physical measurement.

5. Method that enables the detection of the textured structure of the road with the image taken from the camera (20) in vehicles according to claim 1 , characterized in that; virtual measurement is carried out using an artificial neural network under the control of the controller.

6. Method that enables the detection of the textured structure of the road with the image taken from the camera (20) in vehicles according to claim 1 , characterized in that; the measured and finalized end points are marked with coordinate-based positioning / location / alignment data and stored on at least one cloud server under the supervision of the controller.

7. Method that enables the detection of the textured structure of the road with the image taken from the camera (20) in vehicles according to claim 6, characterized in that; the final points measured and finalized under the supervision of the controller are marked with coordinate-based positioning / location / alignment data and the camera (20) evaluates the factors such as buildings, objects, traffic signs, etc. in the environment and localizes the vehicle and the measurement and stores them on the cloud.

Images