Pavement disease positioning and size calculation method and system

Abstract

The invention discloses a pavement disease positioning and size calculation method and system, and relates to the technical field of road disease detection.The method comprises the steps that the rolling angle of a pavement image is corrected to obtain a corrected image, and the position and shape coordinates of a pavement disease in an image coordinate system of the corrected image are determined; determining the position and shape coordinates of the pavement disease in the overlook coordinate system according to the position and shape coordinates of the pavement disease in the image coordinate system of the corrected image by adopting a perspective transformation matrix of the image coordinate system of the corrected image and the overlook coordinate system; determining the position of the pavement disease in the world coordinate system according to the position coordinate of the pavement disease in the overlook coordinate system by adopting the conversion relation between the overlook coordinate system and the world coordinate system; and calculating the size of the pavement disease in the overlook coordinate system according to the shape coordinates of the pavement disease in the overlook coordinate system. The perspective transformation relation from the image coordinate system to the overlook coordinate system is corrected in real time, and then the accuracy of pavement disease size and positioning data calculation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of road defect detection, and in particular to a method and system for locating and calculating the size of road defects. Background Art

[0002] Slow traffic refers to the modes of travel for non-motor vehicles and pedestrians, mainly including walking, cycling, scooters, etc. Compared with motor vehicles, slow traffic is more environmentally friendly and healthy, and can reduce traffic congestion and air pollution. Slow lanes are mainly road facilities for non-motor vehicles and pedestrians to travel, and are an important part of urban transportation infrastructure. With the rapid development of urbanization, slow traffic infrastructure has also been rapidly developed and constructed. After the peak of construction, the management and maintenance of slow traffic has become increasingly prominent.

[0003] Similar to motor vehicle lanes, after a certain number of years of service, slow lanes will develop various defects and diseases under the influence of multiple factors such as temperature, rain, and load. Minor diseases will affect the appearance of the slow lanes, while serious diseases will not only affect the appearance, but also the safety of slow traffic. At present, common slow lane diseases include pavement damage on non-motor vehicle lanes, damage to motor vehicle and non-motor vehicle guardrails, cracking of blocks on the sidewalk, missing blocks, and subsidence. In addition to diseases, there are some incidents on slow lanes that need to be discovered in time, such as missing traffic markings, random parking of non-motor vehicles, occupation of blind paths, missing or damaged manhole covers, etc. In general, the diseases and incidents on slow lanes are characterized by many types, large numbers, and wide distribution.

[0004] To improve the efficiency of solving diseases or incidents on slow lanes, the most important thing is to improve the efficiency of discovering related diseases or incidents. In other words, in addition to timely discovering and reporting related diseases, it is also necessary to accurately describe the diseases, including the location of the disease, the type of disease, the size of the disease and other information. At present, the main reporting method for diseases and incidents on slow lanes is manual foot patrol, that is, the inspectors discover and report problems on the road. This method organizes relevant inspectors to perform inspection tasks on the road regularly, and records and reports them through mobile phones after discovering the target. Although manual foot patrol is currently the most mainstream method, it has problems such as slow speed, large subjective influence of inspectors, and inspectors cannot describe the type or location of the disease in detail or accurately. This method has high requirements on the number and quality of inspectors.

[0005] With the rapid development of deep learning and artificial intelligence technology, some solutions are currently available: install sensors such as perception cameras on slow-moving vehicles or pedestrians, use cameras to collect images of slow lanes, and then use relevant artificial intelligence algorithms to identify diseases or events on slow lanes. Although this type of method improves the efficiency of inspections, it also has some problems: to calculate the size of the disease in the collected image, it is necessary to establish a perspective transformation relationship between image coordinates and world coordinates, and use the perspective transformation relationship to calculate the size of the disease in the image in the corresponding world coordinates. Changes in the perspective transformation relationship will affect the accuracy of the calculation results. If the camera keeps the angle and height unchanged, the perspective transformation relationship will not change, and will not affect the size calculation; however, in the actual driving process of battery vehicles and non-motor vehicles, the camera will shake due to uneven road conditions, and when the vehicle turns, the camera roll angle will change. These factors will cause the camera's perspective change relationship to change, resulting in changes in the conversion relationship between "image coordinates" and "world coordinates", which will lead to inaccurate calculations of disease size and target-level positioning, thereby affecting subsequent data analysis. Summary of the invention

[0006] The purpose of this application is to provide a method and system for locating and calculating the size of road defects, which can reduce the impact of camera shake and rolling on the perspective transformation relationship when the vehicle is driving, thereby improving the accuracy of coordinate transformation, thereby improving the accuracy of determining the defect size and target-level positioning calculation based on the coordinate transformation relationship.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a method for locating and calculating the size of pavement defects, comprising:

[0009] The road surface image is collected by the camera at every accumulated distance;

[0010] Performing target recognition on the road surface image to determine road surface damage in the road surface image;

[0011] Correcting the roll angle of the road surface image to obtain a corrected image, and determining the position coordinates and shape coordinates of the road surface defect in the image coordinate system of the corrected image;

[0012] Determine the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image; the coordinate origin of the overhead coordinate system is the antenna position of the navigation positioning receiver, the positive direction of the X axis is the vehicle forward direction, the positive direction of the Y axis is perpendicular to the vehicle forward direction, and the negative direction of the Z axis is the gravity direction; the camera and the antenna of the navigation positioning receiver are both arranged on the vehicle;

[0013] Using the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system, the position coordinates and shape coordinates of the road surface disease in the overhead coordinate system are determined according to the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image;

[0014] Using the conversion relationship between the overhead coordinate system and the world coordinate system, the position of the pavement defect in the world coordinate system is determined according to the position coordinates of the pavement defect in the overhead coordinate system;

[0015] According to the shape coordinates of the road surface disease in the overhead coordinate system, the size of the road surface disease in the overhead coordinate system is calculated.

[0016] Optionally, the roll angle of the road surface image is corrected to obtain a corrected image, and the position coordinates and shape coordinates of the road surface defect in the image coordinate system of the corrected image are determined, specifically including:

[0017] The roll angle of the road surface image is identified using a roll angle recognition model to determine a current roll angle; the current roll angle is the angle required for the road surface image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model;

[0018] Taking the center point of the road surface image as the center of the circle, rotating the road surface image by the current roll angle to obtain a corrected image;

[0019] The position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image are determined according to the corrected image.

[0020] Optionally, determining the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image specifically includes:

[0021] Calibrate the camera to determine the intrinsic parameter matrix, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle, and the calibrated pitch angle;

[0022] Calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image using a vanishing point recognition model; the vanishing point recognition model is a pre-trained deep neural network;

[0023] Linear fitting is performed on the position coordinates of N square corner points of multiple pairs of checkerboard calibration cloths in the image coordinate system of the rotated calibration image and the calibrated pitch angles to obtain an empirical function;

[0024] Taking the position coordinates of the vanishing point in the image coordinate system of the corrected image as input, and using an empirical function to calculate the current pitch angle;

[0025] A rotation matrix is ​​constructed according to the current pitch angle, the calibrated heading angle and the current roll angle; the rotation matrix is ​​R:

[0026]

[0027] Among them, α is the current pitch angle, β is the calibrated heading angle, and γ is the current roll angle;

[0028] Determine the position coordinates of N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image according to the intrinsic parameter matrix, the rotation matrix and the calibrated translation vector;

[0029] Using the perspective transformation formula [x wn ,y wn ,0]=M[u' n ,v' n ], determining the perspective transformation matrix between the image coordinate system of the corrected image and the overhead coordinate system;

[0030] Among them, [x wn ,y wn ,0] are the position coordinates of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, x wn is the X-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system, wn is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, M is the perspective transformation matrix, [u' n ,v' n ] is the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, u' n is the U-axis coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n The V-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image.

[0031] Optionally, the camera is calibrated to determine the intrinsic parameter matrix, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle, and the calibrated pitch angle, specifically including:

[0032] Calculate the camera's intrinsic parameter matrix;

[0033] Place the checkerboard calibration cloth horizontally on a flat road surface, collect the checkerboard calibration cloth image, and obtain a calibration image;

[0034] According to the calibration image, determining the position coordinates of N square corner points of the checkerboard calibration cloth in the overhead coordinate system;

[0035] The roll angle of the calibration image is identified using a roll angle recognition model to determine a calibration roll angle; the calibration roll angle is the angle required for the calibration image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model;

[0036] Taking the center point of the calibration image as the center of the circle, rotating the calibration image by the current roll angle to obtain a rotated calibration image;

[0037] According to the rotated calibration image, determine the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image;

[0038] The perspective N-point projection algorithm is used to calculate the calibrated translation vector, calibrated heading angle and calibrated pitch angle.

[0039] Optionally, using a vanishing point recognition model to calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image specifically includes:

[0040] Use a pre-trained deep neural network to determine the position coordinates of the vanishing point in the image coordinate system of the corrected image, or use a pre-trained deep neural network to identify feature lines in the corrected image, calculate the intersection points of the feature lines, and use the position coordinates of the intersection points of the feature lines in the image coordinate system of the corrected image as the position coordinates of the vanishing point in the image coordinate system of the corrected image.

[0041] Optionally, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image are determined by the following formula:

[0042]

[0043] Among them, K i is the internal parameter matrix, rvec new is the rotation matrix, t vec is the calibrated translation vector, x wn is the X-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system, wn is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, u' n is the U-axis coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n is the V-axis coordinate of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, Z c The z-axis coordinates of the N square corner points of the chessboard calibration cloth in the camera coordinate system.

[0044] Optionally, after calculating the size of the pavement defect in the overhead coordinate system according to the shape coordinates of the pavement defect in the overhead coordinate system, the pavement defect location and size calculation method further includes:

[0045] Uploading the road surface image and uploaded data to a cloud server via a mobile Internet network; the uploaded data includes the size of the road surface defect in the overhead coordinate system, the position of the road surface defect in the world coordinate system, and the acquisition time;

[0046] When the cloud server receives new uploaded data, it queries the disease database to see whether there is uploaded data with a similarity greater than a threshold with the new uploaded data;

[0047] If there is uploaded data whose similarity with the new uploaded data is greater than the threshold, the new uploaded data will be marked as repeated identification and entered into the disease database; if there is no uploaded data whose similarity with the new uploaded data is greater than the threshold, the new uploaded data will be marked as the first identification and entered into the disease database.

[0048] In a second aspect, the present application provides a road surface disease location and size calculation system, comprising:

[0049] An image acquisition module, used for acquiring road surface images through a camera at every accumulated distance;

[0050] A recognition module, used to perform target recognition on the road surface image and determine road surface damage in the road surface image;

[0051] A correction module, used to correct the roll angle of the road surface image to obtain a corrected image, and determine the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image;

[0052] A perspective transformation matrix determination module is used to determine the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image; the coordinate origin of the overhead coordinate system is the antenna position of the navigation positioning receiver, the positive direction of the X axis is the vehicle forward direction, the positive direction of the Y axis is perpendicular to the vehicle forward direction, and the negative direction of the Z axis is the gravity direction; the camera and the antenna of the navigation positioning receiver are both arranged on the vehicle;

[0053] A coordinate determination module, for determining the position coordinates and shape coordinates of the road surface disease in the above-mentioned overhead coordinate system according to the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image by using the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system;

[0054] A position determination module, for determining the position of the pavement defect in the world coordinate system according to the position coordinates of the pavement defect in the above-mentioned coordinate system by using the conversion relationship between the overhead coordinate system and the world coordinate system;

[0055] The size calculation module is used to calculate the size of the road surface disease in the overhead coordinate system according to the shape coordinates of the road surface disease in the overhead coordinate system.

[0056] Optionally, the correction module includes:

[0057] A roll angle recognition submodule, used to use a roll angle recognition model to recognize the roll angle of the road surface image and determine a current roll angle; the current roll angle is the angle required for the road surface image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model;

[0058] A rotation submodule, used to rotate the road surface image by the current roll angle with the center point of the road surface image as the center of the circle to obtain a corrected image;

[0059] The coordinate determination submodule is used to determine the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image according to the corrected image.

[0060] Optionally, the perspective transformation matrix determination module includes:

[0061] The calibration submodule is used to calibrate the camera, determine the position coordinates of the internal parameter matrix, the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle and the calibrated pitch angle;

[0062] A vanishing point recognition submodule, configured to calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image using a vanishing point recognition model; the vanishing point recognition model is a pre-trained deep neural network;

[0063] The fitting submodule is used to perform linear fitting on the position coordinates of N square corner points of multiple pairs of checkerboard calibration cloths in the image coordinate system of the rotated calibration image and the calibrated pitch angles to obtain an empirical function;

[0064] A pitch angle determination submodule, used to take the position coordinates of the vanishing point in the image coordinate system of the corrected image as input, and calculate the current pitch angle using an empirical function;

[0065] The rotation matrix construction submodule is used to construct a rotation matrix according to the current pitch angle, the calibrated heading angle and the current roll angle; the rotation matrix is ​​R:

[0066]

[0067] Among them, α is the current pitch angle, β is the calibrated heading angle, and γ is the current roll angle;

[0068] A corner point coordinate determination submodule is used to determine the position coordinates of N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image according to the intrinsic parameter matrix, the rotation matrix and the calibrated translation vector;

[0069] The transformation matrix determination submodule is used to adopt the perspective transformation formula [x wn ,y wn ,0]=M[u' n ,v' n ], determine the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system; where [x wn ,y wn ,0] are the position coordinates of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, x wn is the X-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system, wn is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, M is the perspective transformation matrix, [u' n ,v' n ] is the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, u' n is the U-axis coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n The V-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image.

[0070] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0071] The present application provides a method and system for locating and calculating the size of road defects. The method uses a roll angle recognition model to correct a road surface image, determines the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system, and uses the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system to determine the position coordinates and shape coordinates of the road surface defect in the overhead coordinate system, thereby determining the position of the road surface defect in the world coordinate system and the size of the road surface defect in the overhead coordinate system. This allows the present application to reduce the impact of camera shake and roll on coordinate conversion during vehicle driving in the scenario of slow lane inspection, thereby improving the calculation accuracy of converting the defect from the "image coordinate system" to the "world coordinate system", and achieving accurate defect positioning and size calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A device relationship diagram of the intelligent inspection equipment provided in one embodiment of the present application;

[0074] Figure 2 A schematic diagram of the installation of a smart inspection device provided in an embodiment of the present application, in which a high-definition camera and an industrial computer are installed in a basket, and the camera is shooting in the forward direction;

[0075] Figure 3 A schematic diagram of the installation of the intelligent inspection device provided in one embodiment of the present application, wherein a high-definition camera and an industrial computer are installed on a rear seat bracket, and the high-definition camera is facing backwards;

[0076] Figure 4 A schematic diagram of a top-down coordinate system provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of a camera extrinsic calibration process according to an embodiment of the present application;

[0078] Figure 6 A schematic diagram of the equipment collection, identification, and upload process of a method for locating and calculating the size of road defects provided in one embodiment of the present application;

[0079] Figure 7 A schematic diagram of data received and stored by a cloud server in a method for locating and calculating the size of pavement defects provided in one embodiment of the present application.

[0080] Figure numerals: high-definition camera—1, industrial computer—2. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] At present, the inspection of slow lanes is mainly done manually, and the inspection of intelligent equipment is mainly for motor vehicle lanes, and its application in slow lanes is still not common. The current inspection of slow lanes mainly based on manual inspection has the following main problems:

[0083] (1) Slow speed and low efficiency: Manual inspections are usually conducted on foot, which is slow and requires reliance on mobile phone applications (Applications, APPs) to collect and report problems. During the collection process, disease information needs to be filled in manually, which is inefficient.

[0084] (2) High labor costs: The distance that can be covered by a single person on foot in a single day is limited. To achieve a higher patrol coverage, more manpower investment is required.

[0085] (3) Easy to miss: Some inspections are conducted by inspectors driving non-motor vehicles or electric vehicles. Although this type of inspection method can increase the inspection speed, it still mainly relies on inspectors to observe the surrounding diseases and collect and report them through the APP. In addition, due to the increase in inspection speed, inspectors are prone to miss diseases during the inspection, especially some diseases that are relatively subtle on the surface.

[0086] (4) Large subjective influence: During foot inspections, the type of disease and the reported information are closely related to the subjective judgment of the inspectors. Different inspectors’ descriptions of reported diseases may have obvious deviations from the actual situation, such as the size of the disease, the severity of the disease, etc., and may even report errors. The same inspector may also have different descriptions of the same disease under different conditions (such as when he is tired or focused).

[0087] At present, there are also some intelligent patrol devices, which install intelligent patrol devices or smart phones on battery vehicles to collect images and perform intelligent recognition during patrols. Although this method improves the efficiency of patrols, it also has some problems:

[0088] (1) Lack of reference objects: Inspection equipment generally only captures one image, and the captured image may lack reference objects around the target, which may increase the difficulty for maintenance personnel to find the disease.

[0089] (2) Inaccurate size calculation: In order to calculate the size of the defect in the captured image, it is necessary to establish a perspective transformation matrix between the image coordinates and the world coordinates. The perspective transformation matrix can be used to calculate the size of the defect in the image in the corresponding world coordinates. However, when electric vehicles and non-motor vehicles are driving, the camera will vibrate due to the uneven road surface, and the camera roll angle will change due to vehicle turning. These two factors will cause the perspective transformation matrix to be different from that during calibration, resulting in inaccurate calculation of the defect size and target-level positioning, which will affect subsequent data analysis.

[0090] (3) Duplicate data reporting: In one cycle, intelligent inspection equipment may inspect a certain section of the road multiple times, which may lead to multiple identification and reporting of a single disease, resulting in duplicate reported data. The location of the reported disease is generally the longitude and latitude of the image. Therefore, when deduplicating diseases, different diseases in the same image may be deduplicated together.

[0091] The existing technical solution for intelligent inspection is to use a perspective transformation algorithm to transform the collected image into a bird's-eye view image, and then calculate the size of the defect in the bird's-eye view coordinate system. However, this technical solution is based on an assumption that the height and angle of the camera basically do not change during the movement of the vehicle. However, the cameras currently installed on electric vehicles are easily affected by bumps and vehicle turns. Although the height of the camera does not change much, the angle of the camera will change significantly, which will cause the perspective transformation relationship of converting image coordinates into bird's-eye view coordinates to change. To reduce the calculation error caused by camera shake, the perspective transformation relationship must be corrected in a timely manner.

[0092] The purpose of this application is to introduce artificial intelligence technology to correct the perspective transformation relationship from the image coordinate system to the overhead coordinate system in real time, reduce the impact of camera shake and rolling on size and positioning calculations when the vehicle is driving, and then improve the accuracy of coordinate conversion to achieve accurate disease location and size calculation.

[0093] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0094] A method for locating and calculating the size of road defects provided in an embodiment of the present application can be applied to intelligent patrol equipment.

[0095] An intelligent inspection device is installed on a battery vehicle, bicycle or motorcycle to collect image and positioning data. The intelligent recognition algorithm is used to identify the disease type of the slow lane in the image, calculate the size of the disease, and the location of the disease in the world coordinate system. It is marked whether the disease is a repeated identification disease and stored in the database. The implementation method is as follows:

[0096] (1) Equipment composition.

[0097] like Figure 1As shown, the equipment composition of intelligent patrol equipment generally includes 2 or more high-definition cameras, an industrial computer and a cloud server. The industrial computer also has a high-precision navigation and positioning receiver, an Internet card, and a certain AI computing power, which can deploy recognition algorithm programs; in terms of power supply source, the intelligent patrol equipment uses direct current power supply and can be installed on an electric vehicle, powered by the power supply of the electric vehicle or the headlight power cord, or installed on a bicycle, using a large-capacity mobile power supply or solar panels to directly power the intelligent patrol equipment; in terms of waterproof performance, the intelligent patrol equipment has good waterproof performance.

[0098] (2) Equipment installation.

[0099] How to install the equipment Figure 2 and Figure 3 As shown, in the installation position of the intelligent inspection equipment, the industrial computer 2 and the high-definition camera 1 can be installed in the basket or trunk of the battery car in front of the vehicle to ensure that the image captured by the camera will not be blocked and the captured image is not skewed. And the equipment is powered by the battery car headlights. One high-definition camera shoots in front of the vehicle or in the back, mainly used to collect slow-moving road surface diseases, and the other high-definition camera shoots to the left or right side of the vehicle, mainly used to collect images around slow-moving road diseases, to assist inspection and maintenance personnel in finding the location of the front image acquisition. After the camera is installed on the vehicle and the angle is adjusted to ensure that it is not blocked, it can be further fixed by welding or gluing to prevent the camera's viewing angle from changing or swinging left and right during the inspection operation.

[0100] (3) The positioning error of the high-precision navigation and positioning receiver used is ≤1m, the frequency of positioning data collection is ≥10Hz, and the x-direction of the navigation receiver is aligned with the vehicle's forward direction; the pixel volume of the high-definition camera is ≥2 million pixels, one of the high-definition cameras collects images directly in front of the vehicle or directly behind the vehicle to capture images of the disease to be identified, and the other high-definition camera collects images to the left or right side of the vehicle. After the camera is installed, use the supporting software to record the positional relationship between the high-definition camera and the navigation and positioning receiver. Please refer to Figure 4 , take the antenna of the navigation and positioning receiver as the coordinate origin, the vehicle's forward direction as the positive direction of the X-axis, the direction perpendicular to the vehicle's forward direction as the positive direction of the Y-axis, and the direction of gravity as the negative direction of the Z-axis, establish a top-down coordinate system, and record the high-definition camera CAM i The distance to the origin of the coordinate system (xx i ,yy i ), xx i CAM for HD Cameras i To the X-axis coordinate of the origin in the top-down coordinate system, yy i CAM for HD Cameras iThe Y-axis coordinate of the origin in the top-down coordinate system, the angle θ between the camera shooting direction and the vehicle's forward direction i , and record HD camera CAM i Height to groundzz i .

[0101] In an exemplary embodiment, the present application proposes a method for locating and calculating the size of road surface defects. In the embodiment of the present application, the method is executed by an industrial computer and is implemented as follows:

[0102] (1) Equipment internal parameter calibration.

[0103] HD Camera CAM i Before installation, internal calibration is required to calculate and obtain the HD camera CAM i The internal parameter matrix K i .

[0104] (2) Equipment external parameter calibration.

[0105] Place the checkerboard calibration cloth horizontally on a flat road surface, collect the checkerboard calibration cloth image, and obtain the calibration image.

[0106] According to the calibration image, the position coordinates of the N square corner points of the checkerboard calibration cloth in the overhead coordinate system are determined.

[0107] The roll angle of the calibration image is identified using a roll angle recognition model to determine the calibration roll angle; the calibration roll angle is the angle required for the calibration image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model.

[0108] The calibration image is rotated by the current roll angle with the center point of the calibration image as the center of the circle to obtain a rotated calibration image.

[0109] According to the rotated calibration image, determine the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image. The origin of the image coordinate system is the center of the image, and the U axis and V axis are parallel to the directions of the adjacent sides of the image.

[0110] The perspective N-point projection algorithm is used to calculate the calibrated translation vector, calibrated heading angle and calibrated pitch angle.

[0111] Specifically, after the camera is installed and fixed, the camera needs to be calibrated with external parameters, that is, find a flat road surface and place a checkerboard calibration cloth horizontally. It is usually a piece of white cloth, and the white cloth is made up of black and white squares of the same size. There are 7×7 squares in the calibration cloth, and each square is 20cm long and black and white. The effective length and width of the checkerboard calibration cloth are not less than 1m. When calibrating, the high-definition camera CAMi The captured image can clearly show the checkerboard calibration cloth, black and white squares and corner points, and the number of pixels occupied by the checkerboard in the image is not less than 15,000 pixels; the N square corner points generally refer to N non-collinear points in the checkerboard calibration cloth, and N ≥ 4.

[0112] The camera extrinsic calibration process is as follows: Figure 5 As shown, using a high-definition camera CAM i Collect an image as a calibration image and record the coordinates of the N square corner points (i.e., the chessboard corner points) in the top-view coordinate system [x wn ,y wn ,0], using the roll angle recognition model, the roll angle of the calibration image is identified, and the model inference regresses the roll angle required to rotate the calibration image back to the normal viewing angle m , and then rotate the image by angle roll with the center point of the image as the center of the circle m , and record the coordinate positions of N chessboard corner points in the roll angle correction image [u n ,v n The perspective-n-point (PnP) algorithm is used to establish the projection relationship between the N square corner points on the checkerboard calibration cloth in the overhead coordinate system and the N image points in the image imaging, and then the high-definition camera CAM is calculated. i The rotation matrix rvec, translation vector tvec, and current camera CAM in the image calibration state i The pitch angle pitch, heading angle yaw, and roll angle roll.

[0113] (3) Equipment collects data.

[0114] The camera collects road images at every accumulated distance.

[0115] Specifically, the equipment collects road images and positioning data during driving. The road image collection is triggered by the accumulated distance, that is, the accumulated distance of the equipment is accumulated from the starting point. When the equipment travels more than m meters, a road image is collected, and the equipment is reset to accumulate the distance from 0 until the next accumulated distance reaches m meters, and then the road image is collected again, and so on. When saving the road image, the positioning data before and after the road image is collected is intercepted by the timestamp, and the interpolation method is used to calculate the positioning data corresponding to the moment the road image is collected. The positioning data includes longitude, latitude, azimuth, speed and other information.

[0116] (4) Analyze and locate data.

[0117] Target recognition is performed on the road surface image to determine road surface damage in the road surface image.

[0118] The roll angle of the road surface image is corrected to obtain a corrected image, and the position coordinates and shape coordinates of the road surface defect in the image coordinate system of the corrected image are determined.

[0119] Determine the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image; the coordinate origin of the overhead coordinate system is the antenna position of the navigation and positioning receiver, the positive direction of the X-axis is the forward direction of the vehicle, the positive direction of the Y-axis is perpendicular to the forward direction of the vehicle, and the negative direction of the Z-axis is the direction of gravity; the camera and the antenna of the navigation and positioning receiver are both set on the vehicle.

[0120] The perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system is used to determine the position coordinates and shape coordinates of the road surface disease in the overhead coordinate system according to the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image.

[0121] The conversion relationship between the overhead coordinate system and the world coordinate system is used to determine the position of the road surface disease in the world coordinate system according to the position coordinates of the road surface disease in the overhead coordinate system.

[0122] According to the shape coordinates of the road surface disease in the overhead coordinate system, the size of the road surface disease in the overhead coordinate system is calculated.

[0123] In an exemplary embodiment, the roll angle of the road surface image is corrected to obtain a corrected image, and the position coordinates and shape coordinates of the road surface defect in the image coordinate system of the corrected image are determined, specifically including:

[0124] The roll angle of the road surface image is identified using a roll angle recognition model to determine a current roll angle; the current roll angle is the angle required for the road surface image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model.

[0125] The road surface image is rotated by the current roll angle with the center point of the road surface image as the center of the circle to obtain a corrected image.

[0126] The position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image are determined according to the corrected image.

[0127] Specifically, the roll angle recognition model is used to identify the roll angle of the collected road image, and the model infers and regresses the roll angle required to rotate the road image back to the normal viewing angle. new , get the roll angle roll of the road image new , and then the collected road image is corrected for roll angle rotation with the center of the road image as the center of the circle, so that the image rotation angle roll newThe detection model is used to identify the target in the collected road surface image, and then the position [u, v] and shape coordinates [u shape ,v shape ]. The detection model is a pre-trained deep neural network.

[0128] In an exemplary embodiment, the perspective transformation matrix M of the road image is calculated by using a deep neural network to calculate the conversion relationship between the currently acquired image coordinates and the top-view coordinates. The vehicle-mounted camera will produce bumps during driving, and the camera pitch angle will change, which will cause the perspective transformation matrix to change, so the perspective transformation matrix of the road image must be corrected. Determining the perspective transformation matrix of the image coordinate system and the top-view coordinate system of the corrected image specifically includes:

[0129] The camera is calibrated to determine the intrinsic parameter matrix, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle, and the calibrated pitch angle.

[0130] The vanishing point recognition model is used to calculate the position coordinates [u vp ,v vp ],u vp is the U-axis coordinate of the vanishing point in the image coordinate system of the corrected image, v vp is the V-axis coordinate of the vanishing point in the image coordinate system of the corrected image; and the vanishing point recognition model is a pre-trained deep neural network.

[0131] The position coordinates of the N square corner points of the multiple pairs of checkerboard calibration cloth in the image coordinate system of the rotated calibration image and the calibrated pitch angle are linearly fitted to obtain the empirical function F(pitch,v vp ). F is a function derived from the experimental regression in advance, which means that when the camera height is constant, the coordinates of the vanishing point on the v axis in the image and the pitch angle of the camera are approximately linearly related. vp ) can be linearly fitted through multiple calibration data (data of equipment external parameter calibration).

[0132] The position coordinates of the vanishing point in the image coordinate system of the corrected image are taken as input, and the current pitch angle is calculated using an empirical function. new .

[0133] Since the camera is fixed on the vehicle, the heading angle of the camera relative to the vehicle does not change, and the change in the height of the camera from the ground is ignored. The rotation matrix is ​​constructed using Euler angles according to the current pitch angle, the calibrated heading angle and the current roll angle; the rotation matrix is ​​R:

[0134]

[0135] Among them, α is the current pitch angle, β is the calibrated heading angle, and γ is the current roll angle.

[0136] According to the intrinsic parameter matrix, the rotation matrix and the calibrated translation vector, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image are determined.

[0137] Using the perspective transformation formula [x wn ,y wn ,0]=M[u' n ,v' n ], determine the perspective transformation matrix between the image coordinate system of the corrected image and the overhead coordinate system. And use this formula to calculate the position coordinates [u,v] and shape coordinates [u shape ,v shape ], the position coordinates [x, y] and shape coordinates [x shape ,y shape ], according to [x shape ,y shape ]Calculate the size of the identified target in the overhead coordinate system.

[0138] Among them, [x wn ,y wn ,0] is the position coordinates of the N square corner points of the chessboard calibration cloth in the top-down coordinate system, M is the perspective transformation matrix, [u' n ,v' n ] are the position coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image.

[0139] In an exemplary embodiment, using a vanishing point recognition model to calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image specifically includes:

[0140] Use a pre-trained deep neural network to determine the position coordinates of the vanishing point in the image coordinate system of the corrected image or use a deep neural network to identify the feature lines in the corrected image, calculate the intersection points of the feature lines, and use the position coordinates of the intersection points of the feature lines in the image coordinate system of the corrected image as the position coordinates of the vanishing point in the image coordinate system of the corrected image. That is, the vanishing point recognition model is a specially trained deep neural network, which can be one of the following two processing methods: 1. Input the captured image into the neural network, and the neural network directly regresses the position of the vanishing point in the image [u vp ,v vp]; 2. Or use a deep neural network to identify feature lines such as lane lines and block edge lines in the image, and find the intersection of the feature lines as the position of the vanishing point in the image [u vp ,v vp ].

[0141] In an exemplary embodiment, the following formula is used to determine the position coordinates [u' n ,v' n ].

[0142]

[0143] Among them, K i is the internal parameter matrix, rvec new is the rotation matrix, tvec is the calibrated translation vector, x wn y is the X-axis coordinates of the N square corner points of the chessboard calibration cloth in the top-view coordinate system; wn u' is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system; n is the U-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image; v' n is the V-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image; c is the z-axis coordinate of the N square corner points of the chessboard calibration cloth in the camera coordinate system, which can be directly obtained by the Euclidean distance between the camera and the corner point. The origin of the camera coordinate system is the focus center of the HD camera, the x-axis and y-axis are parallel to the adjacent sides of the image, and the z-axis is the optical axis direction of the HD camera.

[0144] Recording HD camera CAM according to camera calibration i The distance to the origin of the coordinate system (xx i ,yy i ), the angle θ between the camera shooting direction and the vehicle's forward direction i , calculate the position of the recognized target in the world coordinate system [lng, lat] in the overhead coordinate system [x, y], and then complete the target-level positioning calculation of the image recognition target, and then save the calculated data locally and upload it to the cloud.

[0145] The intelligent patrol equipment of the present application is easy to install. The intelligent patrol equipment uses edge computing power to achieve intelligent identification, which can realize fast, accurate and quantitative perception of slow-moving road surface diseases and events, improve patrol speed, accuracy, and improve patrol efficiency.

[0146] In view of the bumpy road conditions in slow-moving traffic, the camera's roll angle and pitch angle are prone to change. The equipment collection, recognition, and upload process is as follows: Figure 6 As shown, the present application uses a roll angle recognition model to identify the roll angle, uses a detection model to identify the target in the collected image, and calculates the position [u, v] and shape coordinates [u shape ,v shape ], use the vanishing point recognition model to calculate the perspective transformation matrix M (i.e., the perspective transformation matrix of the current image) between the image coordinate system and the bird’s-eye view coordinate system of the corrected image, and then calculate the position [x, y] and shape [x shape ,y shape ], according to [x shape ,y shape ] Calculate the size of the road surface disease (i.e., the identification target) in the overhead coordinate system, and then use the relationship between the overhead coordinate system and the world coordinate system to calculate the position of the road surface disease (i.e., the identification target) in the world coordinate system [lng, lat]. By quickly calibrating the camera extrinsic parameters and using the intelligent recognition algorithm, the captured image can be corrected for the extrinsic parameters in real time, the size calculation of the road surface disease can be optimized, and each road surface disease in the image can be positioned at the target level. Thereby, in the scenario of slow lane inspection, the calculation accuracy of the disease from the "image coordinate system" to the "world coordinate system" is improved, and accurate disease location and size calculation are achieved. In addition, on this basis, the present application can also match the same disease inspected multiple times according to the low-error disease location and size calculation, so as to reduce repeated data reporting. The implementation method is as follows:

[0147] (5) Upload images, receive data on the cloud and store them in the database.

[0148] The road surface image and uploaded data are uploaded to the cloud server via the mobile Internet; the uploaded data includes the size of the road surface disease in the overhead coordinate system, the position of the road surface disease in the world coordinate system, and the acquisition time.

[0149] When the cloud server receives new uploaded data, it queries the disease database to see whether there is uploaded data with a similarity greater than a threshold with the new uploaded data.

[0150] If there is uploaded data whose similarity with the new uploaded data is greater than the threshold, the new uploaded data will be marked as repeated identification and entered into the disease database; if there is no uploaded data whose similarity with the new uploaded data is greater than the threshold, the new uploaded data will be marked as the first identification and entered into the disease database.

[0151] Specifically, the data identified by the smart patrol device will be uploaded to the cloud server. The data receiving service is deployed on the cloud server. After the smart patrol device collects and identifies the target, it will upload the collected and analyzed data to the cloud server through the mobile Internet network. The uploaded data includes information such as the type of identification, size, target-level positioning, and collection time.

[0152] The cloud server receives and stores data such as Figure 7 As shown, the cloud server is provided with a disease database DB, and when the cloud server receives new identification data A, it queries whether there is similar data in the DB according to information such as category, longitude and latitude, size, date, etc.: if there is identification data with similar location, same or similar category, and similar size, A will be marked as a repeated identification disease and entered into the database DB; otherwise, A will be marked as the first identification (i.e., non-repeated disease) and entered into the database DB.

[0153] The identification results are uploaded to the cloud and can be merged and deduplicated based on the identification type, size, location, and time. The collected identification has greater use value: on the one hand, it can avoid data duplication caused by high-frequency inspections, and can effectively eliminate duplicate data when performing task dispatching and statistical analysis, thereby improving management and maintenance efficiency; on the other hand, the data obtained from repeated inspections can also be used for continuous tracking of diseases and can be used for early warning of disease deterioration.

[0154] This application proposes a method for locating and calculating the size of road defects. An intelligent inspection device is installed on an electric vehicle / non-motor vehicle to realize the intelligent identification of slow-moving road defects, and an algorithm is used to optimize the problems of defect location, size identification, and duplicate data reporting, thereby improving the efficiency and accuracy of data collection.

[0155] Based on the same inventive concept, the embodiment of the present application also provides a road surface disease location and size calculation system for implementing the road surface disease location and size calculation method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more road surface disease location and size calculation system embodiments provided below can refer to the limitations of the road surface disease location and size calculation method above, and will not be repeated here.

[0156] In an exemplary embodiment, a pavement defect location and size calculation system is provided, comprising:

[0157] The image acquisition module is used to acquire road surface images through a camera at every accumulated distance.

[0158] The recognition module is used to perform target recognition on the road surface image and determine the road surface damage in the road surface image.

[0159] The correction module is used to correct the roll angle of the road surface image to obtain a corrected image, and determine the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image.

[0160] A perspective transformation matrix determination module is used to determine the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image; the coordinate origin of the overhead coordinate system is the antenna position of the navigation and positioning receiver, the positive direction of the X-axis is the vehicle's forward direction, the positive direction of the Y-axis is perpendicular to the vehicle's forward direction, and the negative direction of the Z-axis is the direction of gravity; the camera and the antenna of the navigation and positioning receiver are both arranged on the vehicle.

[0161] The coordinate determination module is used to use the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system to determine the position coordinates and shape coordinates of the road surface disease in the overhead coordinate system according to the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image.

[0162] The position determination module is used to determine the position of the road surface disease in the world coordinate system according to the position coordinates of the road surface disease in the above-mentioned coordinate system by using the conversion relationship between the overhead coordinate system and the world coordinate system.

[0163] The size calculation module is used to calculate the size of the road surface disease in the overhead coordinate system according to the shape coordinates of the road surface disease in the overhead coordinate system.

[0164] As an optional implementation, the correction module includes:

[0165] The roll angle recognition submodule is used to use a roll angle recognition model to identify the roll angle of the road surface image and determine the current roll angle; the current roll angle is the angle required for the road surface image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model.

[0166] The rotation submodule is used to rotate the road surface image by the current roll angle with the center point of the road surface image as the center of the circle to obtain a corrected image.

[0167] The coordinate determination submodule is used to determine the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image according to the corrected image.

[0168] As an optional implementation, the perspective transformation matrix determination module includes:

[0169] The calibration submodule is used to calibrate the camera, determine the internal parameter matrix, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle and the calibrated pitch angle.

[0170] The vanishing point recognition submodule is used to calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image using a vanishing point recognition model; the vanishing point recognition model is a pre-trained deep neural network.

[0171] The fitting submodule is used to perform linear fitting on the position coordinates of N square corner points of multiple pairs of checkerboard calibration cloths in the image coordinate system of the rotated calibration image and the calibrated pitch angles to obtain an empirical function;

[0172] The pitch angle determination submodule is used to take the position coordinates of the vanishing point in the image coordinate system of the corrected image as input and calculate the current pitch angle using an empirical function.

[0173] The rotation matrix construction submodule is used to construct a rotation matrix according to the current pitch angle, the calibrated heading angle and the current roll angle; the rotation matrix is ​​R:

[0174]

[0175] Among them, α is the current pitch angle, β is the calibrated heading angle, and γ is the current roll angle.

[0176] A corner point coordinate determination submodule is used to determine the position coordinates of N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image according to the intrinsic parameter matrix, the rotation matrix and the calibrated translation vector;

[0177] The transformation matrix determination submodule is used to adopt the perspective transformation formula [x wn ,y wn ,0]=M[u' n ,v' n ], determine the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system; where [x wn ,y wn ,0] are the position coordinates of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, x wn is the X-axis coordinate of the N square corner points of the checkerboard calibration cloth in the above-mentioned top-view coordinate system, y wn is the Y-axis coordinate of the N square corner points of the checkerboard calibration cloth in the above-mentioned top-view coordinate system, M is the perspective transformation matrix, [u' n ,v' n ] is the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, u' n is the U-axis coordinate of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n are the V-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for locating and calculating the size of pavement defects, characterized in that: The pavement disease location and size calculation method includes: The road surface image is collected by the camera at every accumulated distance; Performing target recognition on the road surface image to determine road surface damage in the road surface image; Correcting the roll angle of the road surface image to obtain a corrected image, and determining the position coordinates and shape coordinates of the road surface defect in the image coordinate system of the corrected image; Determine the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image; the coordinate origin of the overhead coordinate system is the antenna position of the navigation positioning receiver, the positive direction of the X axis is the vehicle forward direction, the positive direction of the Y axis is perpendicular to the vehicle forward direction, and the negative direction of the Z axis is the gravity direction; the camera and the antenna of the navigation positioning receiver are both arranged on the vehicle; Using the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system, the position coordinates and shape coordinates of the road surface disease in the overhead coordinate system are determined according to the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image; Using the conversion relationship between the overhead coordinate system and the world coordinate system, the position of the pavement defect in the world coordinate system is determined according to the position coordinates of the pavement defect in the overhead coordinate system; According to the shape coordinates of the road surface disease in the overhead coordinate system, the size of the road surface disease in the overhead coordinate system is calculated.

2. The method for locating and calculating the size of pavement defects according to claim 1, characterized in that: Correcting the roll angle of the road surface image to obtain a corrected image, and determining the position coordinates and shape coordinates of the road surface defect in the image coordinate system of the corrected image, specifically includes: The roll angle of the road surface image is identified using a roll angle recognition model to determine a current roll angle; the current roll angle is the angle required for the road surface image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model; Taking the center point of the road surface image as the center of the circle, rotating the road surface image by the current roll angle to obtain a corrected image; The position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image are determined according to the corrected image.

3. The method for locating and calculating the size of pavement defects according to claim 2, characterized in that: Determining the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image specifically includes: Calibrate the camera to determine the intrinsic parameter matrix, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle, and the calibrated pitch angle; Calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image using a vanishing point recognition model; the vanishing point recognition model is a pre-trained deep neural network; Linear fitting is performed on the position coordinates of N square corner points of multiple pairs of checkerboard calibration cloths in the image coordinate system of the rotated calibration image and the calibrated pitch angles to obtain an empirical function; Taking the position coordinates of the vanishing point in the image coordinate system of the corrected image as input, and using an empirical function to calculate the current pitch angle; A rotation matrix is ​​constructed according to the current pitch angle, the calibrated heading angle and the current roll angle; the rotation matrix is ​​R: Among them, α is the current pitch angle, β is the calibrated heading angle, and γ is the current roll angle; Determine the position coordinates of N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image according to the intrinsic parameter matrix, the rotation matrix and the calibrated translation vector; Using the perspective transformation formula [x wn ,y wn ,0]=M[u' n ,v' n ], determining the perspective transformation matrix between the image coordinate system of the corrected image and the overhead coordinate system; Among them, [x wn ,y wn ,0] are the position coordinates of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, x wn is the X-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system, wn is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, M is the perspective transformation matrix, [u' n ,v' n ] is the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, u' n is the U-axis coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n The V-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image.

4. The method for locating and calculating the size of pavement defects according to claim 3, characterized in that: Calibrate the camera to determine the intrinsic parameter matrix, the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle, and the calibrated pitch angle, including: Calculate the camera's intrinsic parameter matrix; Place the checkerboard calibration cloth horizontally on a flat road surface, collect the checkerboard calibration cloth image, and obtain a calibration image; According to the calibration image, determining the position coordinates of N square corner points of the checkerboard calibration cloth in the overhead coordinate system; The roll angle of the calibration image is identified using a roll angle recognition model to determine a calibration roll angle; the calibration roll angle is the angle required for the calibration image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model; Taking the center point of the calibration image as the center of the circle, rotating the calibration image by the current roll angle to obtain a rotated calibration image; According to the rotated calibration image, determine the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image; The perspective N-point projection algorithm is used to calculate the calibrated translation vector, calibrated heading angle and calibrated pitch angle.

5. The method for locating and calculating the size of pavement defects according to claim 3, characterized in that: Calculating the position coordinates of the vanishing point in the image coordinate system of the corrected image using the vanishing point recognition model specifically includes: Use a pre-trained deep neural network to determine the position coordinates of the vanishing point in the image coordinate system of the corrected image, or use a pre-trained deep neural network to identify feature lines in the corrected image, calculate the intersection points of the feature lines, and use the position coordinates of the intersection points of the feature lines in the image coordinate system of the corrected image as the position coordinates of the vanishing point in the image coordinate system of the corrected image.

6. The method for locating and calculating the size of pavement defects according to claim 3, characterized in that: The following formula is used to determine the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image: Among them, K i is the internal parameter matrix, rvec new is the rotation matrix, t vec is the calibrated translation vector, x wn is the X-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system, wn is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, u' n is the U-axis coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n is the V-axis coordinate of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, Z c The z-axis coordinates of the N square corner points of the chessboard calibration cloth in the camera coordinate system.

7. The method for locating and calculating the size of pavement defects according to claim 1, characterized in that: After calculating the size of the pavement defect in the overhead coordinate system according to the shape coordinates of the pavement defect in the overhead coordinate system, the pavement defect location and size calculation method further includes: Uploading the road surface image and uploaded data to a cloud server via a mobile Internet network; the uploaded data includes the size of the road surface defect in the overhead coordinate system, the position of the road surface defect in the world coordinate system, and the acquisition time; When the cloud server receives new uploaded data, it queries the disease database to see whether there is uploaded data with a similarity greater than a threshold with the new uploaded data; If there is uploaded data whose similarity with the new uploaded data is greater than the threshold, the new uploaded data will be marked as repeated identification and entered into the disease database; if there is no uploaded data whose similarity with the new uploaded data is greater than the threshold, the new uploaded data will be marked as the first identification and entered into the disease database.

8. A road surface disease location and size calculation system, characterized in that: The pavement disease location and size calculation system comprises: An image acquisition module, used for acquiring road surface images through a camera at every accumulated distance; A recognition module, used to perform target recognition on the road surface image and determine road surface damage in the road surface image; A correction module, used to correct the roll angle of the road surface image to obtain a corrected image, and determine the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image; A perspective transformation matrix determination module is used to determine the perspective transformation matrix of the image coordinate system and the overhead coordinate system of the corrected image; the coordinate origin of the overhead coordinate system is the antenna position of the navigation positioning receiver, the positive direction of the X axis is the vehicle forward direction, the positive direction of the Y axis is perpendicular to the vehicle forward direction, and the negative direction of the Z axis is the gravity direction; the camera and the antenna of the navigation positioning receiver are both arranged on the vehicle; A coordinate determination module, for determining the position coordinates and shape coordinates of the road surface disease in the above-mentioned overhead coordinate system according to the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image by using the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system; A position determination module, for determining the position of the pavement defect in the world coordinate system according to the position coordinates of the pavement defect in the above-mentioned coordinate system by using the conversion relationship between the overhead coordinate system and the world coordinate system; The size calculation module is used to calculate the size of the road surface disease in the overhead coordinate system according to the shape coordinates of the road surface disease in the overhead coordinate system.

9. The pavement defect location and size calculation system according to claim 8, characterized in that: The correction module comprises: A roll angle recognition submodule, used to use a roll angle recognition model to recognize the roll angle of the road surface image and determine a current roll angle; the current roll angle is the angle required for the road surface image to rotate back to a normal viewing angle; the roll angle recognition model is a pre-trained deep neural network model; A rotation submodule, used to rotate the road surface image by the current roll angle with the center point of the road surface image as the center of the circle to obtain a corrected image; The coordinate determination submodule is used to determine the position coordinates and shape coordinates of the road surface disease in the image coordinate system of the corrected image according to the corrected image.

10. The road surface defect location and size calculation system according to claim 9, characterized in that: The perspective transformation matrix determination module comprises: The calibration submodule is used to calibrate the camera, determine the position coordinates of the internal parameter matrix, the N square corner points of the checkerboard calibration cloth in the image coordinate system of the rotated calibration image, the calibrated translation vector, the calibrated heading angle and the calibrated pitch angle; A vanishing point recognition submodule, configured to calculate the position coordinates of the vanishing point in the image coordinate system of the corrected image using a vanishing point recognition model; the vanishing point recognition model is a pre-trained deep neural network; The fitting submodule is used to perform linear fitting on the position coordinates of N square corner points of multiple pairs of checkerboard calibration cloths in the image coordinate system of the rotated calibration image and the calibrated pitch angles to obtain an empirical function; A pitch angle determination submodule, used to take the position coordinates of the vanishing point in the image coordinate system of the corrected image as input, and calculate the current pitch angle using an empirical function; The rotation matrix construction submodule is used to construct a rotation matrix according to the current pitch angle, the calibrated heading angle and the current roll angle; the rotation matrix is ​​R: Among them, α is the current pitch angle, β is the calibrated heading angle, and γ is the current roll angle; A corner point coordinate determination submodule is used to determine the position coordinates of N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image according to the intrinsic parameter matrix, the rotation matrix and the calibrated translation vector; The transformation matrix determination submodule is used to adopt the perspective transformation formula [x wn ,y wn ,0]=M[u' n ,v' n ], determine the perspective transformation matrix of the image coordinate system of the corrected image and the overhead coordinate system; where [x wn ,y wn ,0] are the position coordinates of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, x wn is the X-axis coordinate of the N square corner points of the chessboard calibration cloth in the top-view coordinate system, wn is the Y-axis coordinate of the N square corner points of the chessboard calibration cloth in the above-mentioned top-view coordinate system, M is the perspective transformation matrix, [u' n ,v' n ] is the position coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, u' n is the U-axis coordinates of the N square corner points of the checkerboard calibration cloth in the image coordinate system of the corrected image, v' n The V-axis coordinates of the N square corner points of the chessboard calibration cloth in the image coordinate system of the corrected image.

Images