Road disease position determination method, system, device and computer equipment

Abstract

This application relates to a method, system, apparatus, and computer device for determining the location of road defects. The method includes: acquiring the vehicle's position and azimuth angle measured by an onboard positioning device, and acquiring digital images and depth images of the road defects captured by an onboard camera; determining a first distance between the road defect and the vehicle based on the digital images and the depth images; correcting the first distance based on the device distance between the onboard positioning device and the onboard camera to obtain a second distance between the road defect and the vehicle; and determining the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance. This method enables high accuracy in determining the location of road defects.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, apparatus and computer equipment for determining the location of road defects. Background Technology

[0002] During the road defect detection process, it is necessary to accurately record the location of the defects so that the affected areas can be tracked, treated, and maintained based on the accurate records.

[0003] In existing technologies, the location of road defects can be determined based on the PNP (Perspective-n-Point) method. This involves photographing the defect area on the road with a camera and calculating the pose of the camera coordinate system relative to the world coordinate system to obtain the location of the defect area in the world coordinate system. However, due to limitations in computational accuracy, the location of defects obtained using the PNP method typically has a large error. According to actual tests, the location error is usually 1-5m on flat, constant-speed roads, and can reach 5-8m on bumpy, uphill or downhill roads.

[0004] Therefore, current technologies for determining the location of road defects have limitations in accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, and computer-readable storage medium that can improve the accuracy of determining the location of road defects in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for determining the location of road defects. The method includes:

[0007] The vehicle position and azimuth angle measured by the vehicle's onboard positioning device are obtained, as well as the digital and depth images obtained by the vehicle's onboard camera from taking pictures of road defects.

[0008] Based on the digital image and the depth image, a first distance between the road defect and the vehicle is determined;

[0009] Based on the device distance between the vehicle positioning device and the vehicle camera, the first distance is corrected to obtain the second distance between the road defect and the vehicle;

[0010] The location of the road defect is determined based on the vehicle's position, the vehicle's azimuth angle, and the second distance.

[0011] In one embodiment, the vehicle-mounted camera includes a first vehicle-mounted camera and a second vehicle-mounted camera, the first vehicle-mounted camera being used to capture the digital image, and the second vehicle-mounted camera being used to capture the depth image; determining the first distance between the road defect and the vehicle based on the digital image and the depth image includes:

[0012] Determine the pixel coordinates of the road defects in the digital image;

[0013] Based on the mapping relationship between the digital image and the depth image, determine the depth coordinates in the depth image corresponding to the pixel coordinates;

[0014] Based on the depth value corresponding to the depth coordinates, a first distance is determined between the road defect and the vehicle.

[0015] In one embodiment, determining the first distance between the road defect and the vehicle based on the depth value corresponding to the depth coordinates includes:

[0016] The depth value corresponding to the depth coordinates is determined as the shooting distance for the vehicle-mounted camera to photograph the road defects;

[0017] Based on the shooting distance and the shooting height of the vehicle-mounted camera, determine the projection distance corresponding to the shooting distance;

[0018] The projection distance corresponding to the shooting distance is determined as the first distance between the road defect and the vehicle.

[0019] In one embodiment, the step of correcting the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain a second distance between the road defect and the vehicle includes:

[0020] The first distance is added to the distance of the device to obtain the second distance between the road defect and the vehicle.

[0021] In one embodiment, the vehicle location includes the vehicle's longitude and latitude coordinates; determining the location of the road defect based on the vehicle location, the vehicle's azimuth angle, and the second distance includes:

[0022] Determine the tangential radius corresponding to the vehicle's latitude coordinates, and determine the horizontal displacement of the road defect relative to the vehicle based on the vehicle's azimuth angle and the second distance;

[0023] Based on the horizontal displacement and the tangential radius, determine the longitude variation value of the road defect relative to the vehicle;

[0024] The longitude coordinates of the vehicle are added to the longitude change value to obtain the longitude coordinates of the road defect.

[0025] In one embodiment, determining the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance further includes:

[0026] The vertical displacement of the road defect relative to the vehicle is determined based on the vehicle's azimuth angle and the second distance.

[0027] Based on the vertical displacement and the preset average radius of the Earth, the latitudinal change value of the road defect relative to the vehicle is determined;

[0028] The latitude coordinates of the vehicle are added to the latitude change value to obtain the latitude coordinates of the road defect.

[0029] Secondly, this application also provides a road defect location determination system. The system includes: a vehicle-mounted positioning device, a vehicle-mounted camera, and a controller; both the vehicle-mounted positioning device and the vehicle-mounted camera are mounted on a vehicle.

[0030] The vehicle positioning device is used to measure the vehicle position and azimuth angle of the vehicle, and send the vehicle position and azimuth angle to the controller;

[0031] The vehicle-mounted camera is used to photograph road defects, obtain digital images and depth images, and send the digital images and depth images to the controller;

[0032] The controller is configured to acquire the vehicle position, the vehicle azimuth angle, the digital image, and the depth image; determine a first distance between the road defect and the vehicle based on the digital image and the depth image; correct the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain a second distance between the road defect and the vehicle; and determine the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance.

[0033] Thirdly, this application also provides a device for determining the location of road defects. The device includes:

[0034] The parameter acquisition module is used to acquire the vehicle position and azimuth angle of the vehicle as measured by the vehicle's on-board positioning device, as well as the digital image and depth image obtained by the vehicle's on-board camera from taking pictures of road defects.

[0035] A distance determination module is used to determine a first distance between the road defect and the vehicle based on the digital image and the depth image;

[0036] A distance correction module is used to correct the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain a second distance between the road defect and the vehicle.

[0037] The location determination module is used to determine the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance.

[0038] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0039] The vehicle position and azimuth angle measured by the vehicle's onboard positioning device are obtained, as well as the digital and depth images obtained by the vehicle's onboard camera from taking pictures of road defects.

[0040] Based on the digital image and the depth image, a first distance between the road defect and the vehicle is determined;

[0041] Based on the device distance between the vehicle positioning device and the vehicle camera, the first distance is corrected to obtain the second distance between the road defect and the vehicle;

[0042] The location of the road defect is determined based on the vehicle's position, the vehicle's azimuth angle, and the second distance.

[0043] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0044] The vehicle position and azimuth angle measured by the vehicle's onboard positioning device are obtained, as well as the digital and depth images obtained by the vehicle's onboard camera from taking pictures of road defects.

[0045] Based on the digital image and the depth image, a first distance between the road defect and the vehicle is determined;

[0046] Based on the device distance between the vehicle positioning device and the vehicle camera, the first distance is corrected to obtain the second distance between the road defect and the vehicle;

[0047] The location of the road defect is determined based on the vehicle's position, the vehicle's azimuth angle, and the second distance.

[0048] The aforementioned method, system, device, computer equipment, and storage medium for determining the location of road defects first acquire the vehicle position and azimuth angle measured by the vehicle-mounted positioning device, and acquire digital and depth images captured by the vehicle-mounted camera. Then, a first distance between the road defect and the vehicle is determined based on the digital and depth images. Next, the first distance is corrected based on the device distance between the vehicle-mounted positioning device and the vehicle-mounted camera to obtain a second distance between the road defect and the vehicle. Finally, the location of the road defect is determined based on the vehicle position, vehicle azimuth angle, and second distance. This allows the first distance determined based on the digital and depth images to be the distance between the road defect and the vehicle-mounted camera. The second distance, obtained by correcting the first distance using the distance between the vehicle-mounted positioning device and the vehicle-mounted camera, is the distance between the road defect and the vehicle-mounted positioning device. Since the vehicle position, vehicle azimuth angle, and second distance are all based on the vehicle-mounted positioning device, determining the location of the road defect based on these factors results in high accuracy.

[0049] Furthermore, by using a TOF camera to capture depth images, real-time measurements can be taken of road defects at a considerable distance. Combined with digital images captured by traffic checkpoint cameras, the distance between road defects and vehicles can be efficiently measured even when the road defects are far away, thereby improving the efficiency of determining the location of road defects. Attached Figure Description

[0050] Figure 1 This is an application environment diagram of the road defect location determination method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a method for determining the location of road defects in one embodiment;

[0052] Figure 3 This is a schematic diagram illustrating the mapping relationship between a digital image and a depth image in one embodiment;

[0053] Figure 4 This is a schematic diagram illustrating the positional relationship between a vehicle and road defects in one embodiment;

[0054] Figure 5 Here is a structural block diagram of a road defect location determination system in one embodiment;

[0055] Figure 6 This is a structural block diagram of a road defect location determination device in one embodiment;

[0056] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The method for determining the location of road defects provided in this application can be applied to, for example... Figure 1 In the application environment shown, vehicle 120 is equipped with an onboard positioning device 130 and an onboard camera 140. Both the onboard positioning device 130 and the onboard camera 140 communicate with the controller 110. The onboard camera 140 captures images of road defects 150. The controller 110 can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server can be a standalone server or a server cluster consisting of multiple servers.

[0059] In one embodiment, such as Figure 2 As shown, a method for determining the location of road defects is provided, which can be applied to... Figure 1 Taking the controller in the example, the following steps are included:

[0060] Step S210: Obtain the vehicle position and azimuth angle measured by the vehicle's onboard positioning device, and obtain the digital image and depth image obtained by the vehicle's onboard camera from taking pictures of road defects.

[0061] Among them, the vehicle positioning equipment can be a vehicle locator or a vehicle positioning terminal, such as a vehicle navigation system, a vehicle GPS (Global Positioning System) receiver, a vehicle Beidou receiver, or a vehicle GNSS (Global Navigation Satellite System) receiver.

[0062] The vehicle location can be the longitude and latitude of the vehicle in the NED (North East Down) coordinate system.

[0063] Among them, the vehicle azimuth angle can be the angle between the vehicle's driving direction and the north axis of the NED coordinate system.

[0064] The vehicle-mounted camera can include a TOF (Time of Flight) camera and a traffic checkpoint camera. Specifically, it can be a TOF camera and a traffic checkpoint camera installed separately on the vehicle, or it can be a single vehicle-mounted camera that integrates the TOF camera and the traffic checkpoint camera.

[0065] In practice, vehicle positioning equipment and vehicle cameras can be installed on the vehicle. The vehicle positioning equipment collects the vehicle's position and azimuth angle and sends them to the controller. The vehicle camera takes pictures of road defects and obtains digital images and depth images, which are also sent to the controller. The controller receives the vehicle's position and azimuth angle sent by the vehicle positioning equipment, as well as the digital images and depth images sent by the vehicle camera, and stores the vehicle's position, azimuth angle, digital images, and depth images.

[0066] For example, a GPS receiver can be installed on the vehicle's roof to locate the vehicle and obtain its latitude and longitude in real time. The GPS receiver can be equipped with dual antennas to obtain the vehicle's azimuth angle in real time. Alternatively, a Time-of-Flight (TOF) camera and a traffic checkpoint camera can be installed on the vehicle's roof in the same location. If the TOF camera and traffic checkpoint camera are not integrated, the field of view of the TOF camera can be adjusted to match that of the traffic checkpoint camera, and both cameras can be controlled to photograph the same damaged area on the road, obtaining a depth image from the TOF camera and a digital image from the traffic checkpoint camera. If the TOF camera and traffic checkpoint camera are integrated, the integrated camera can be used directly to photograph the damaged area on the road, obtaining both depth and digital images. The controller can obtain the vehicle's latitude and longitude, azimuth angle, and the depth image and digital image from the TOF camera and traffic checkpoint camera, respectively.

[0067] Step S220: Determine the first distance between the road defect and the vehicle based on the digital image and the depth image.

[0068] In this context, a digital image can be an image represented by a two-dimensional set of numbers, where each number corresponds to a pixel.

[0069] The depth image can be an image containing depth information, where the depth information can be the shooting distance of the corresponding pixel.

[0070] The first distance can be the distance between road defects and vehicles identified through image recognition.

[0071] In practice, the controller can identify the location of the defect area in the digital image. Based on the mapping relationship between the digital image and the depth image, the controller determines the location of the defect area in the depth image from the location of the defect area in the digital image and obtains the depth information corresponding to the location of the defect area in the depth image. The depth information represents the distance between the defect area on the road and the on-board camera on the vehicle. Therefore, the first distance between the road defect and the vehicle can be determined based on the depth information.

[0072] For example, the TOF camera and traffic checkpoint camera can be pre-calibrated, and the mapping relationship between the depth image captured by the TOF camera and the digital image captured by the traffic checkpoint camera can be calculated. This mapping relationship can be, but is not limited to, a mapping relationship of size and / or angle. By identifying the diseased area in the digital image and obtaining the pixel coordinates (P1x, P1y) of the upper left corner and (P2x, P2y) of the lower right corner of the diseased area, the pixel coordinates (P3x, P3y) of the center point of the diseased area can be...

[0073] P3x = P1x + (P2x - P1x) / 2,

[0074] P3y = P1y + (P2y - P1y) / 2.

[0075] Figure 3 A schematic diagram illustrating the mapping relationship between a digital image and a depth image is provided. Based on... Figure 3 The digital image captured by the traffic checkpoint camera has a pixel size of 2464×2056, while the depth image captured by the TOF camera has a pixel size of 1750×480. The pre-calculated mapping relationship between the depth image and the digital image is x'=x / 2464×1750, y'=y / 2056×480, where x' and y' are the x and y coordinates of a pixel in the depth image, respectively, and x and y are the x and y coordinates of a pixel in the digital image, respectively. Therefore, mapping the pixel coordinates (P3x, P3y) of the center point of the disease area in the digital image onto the depth image yields the center point of the disease area in the depth image. The pixel coordinates (P3x / 2464×1750, P3y / 2056×480) of the point are, in the depth image, the depth value D corresponding to the pixel coordinates (P3x / 2464×1750, P3y / 2056×480) is actually the distance between the center point of the road defect area and the TOF camera on the vehicle. The horizontal distance between the center point of the road defect area and the TOF camera on the vehicle can be determined based on the depth value D corresponding to the pixel coordinates (P3x / 2464×1750, P3y / 2056×480), and the determined horizontal distance is used as the first distance between the road defect area and the vehicle.

[0076] Step S230: Based on the device distance between the vehicle positioning device and the vehicle camera, the first distance is corrected to obtain the second distance between the road defect and the vehicle.

[0077] The second distance can be the corrected distance between the road defect and the vehicle.

[0078] In practice, the distance between the vehicle-mounted positioning device and the vehicle-mounted camera can be measured in advance. The first distance can be corrected using the device distance to obtain the distance between the road defect area and the vehicle-mounted positioning device. The distance between the road defect area and the vehicle-mounted positioning device can be used as the second distance between the road defect and the vehicle.

[0079] Figure 4 A schematic diagram illustrating the positional relationship between vehicles and road defects is provided. Based on... Figure 4 A represents the vehicle positioning device, B represents the vehicle camera (including TOF camera and traffic checkpoint camera), and C represents the road defect. The distance between the vehicle positioning device A and the vehicle camera B can be measured in advance using a measuring tape. The first distance between the road defect C and the vehicle camera B is obtained through step S220. The device distance is added to the first distance, and the first distance is corrected to obtain the second distance between the road defect and the vehicle. The second distance is actually the distance between the road defect C and the vehicle positioning device A.

[0080] Step S240: Determine the location of the road defect based on the vehicle position, vehicle azimuth angle, and second distance.

[0081] In practice, the change in the position of the damaged area on the road relative to the vehicle can be determined based on the vehicle's azimuth angle and the second distance. Based on the vehicle's position and the determined change value, the position of the damaged area on the road can be obtained.

[0082] For example, according to Figure 4 The vehicle positioning device A measures the vehicle's latitude and longitude coordinates as (longA, latA), the vehicle's azimuth angle is θ, and the second distance between the road defect C and the vehicle is d. Therefore, the formula for calculating the latitude and longitude coordinates (longC, latC) of the road defect C is:

[0083] longC=longA+d*sinθ / [ARC*cos(latA)*2π / 360],

[0084] latC=latA+d*cosθ / (ARC*2π / 360),

[0085] Where ARC is the average radius of the equatorial circle, approximately 6,371,393 meters.

[0086] The aforementioned method for determining the location of road defects involves first acquiring the vehicle's position and azimuth angle measured by an onboard positioning device, and then acquiring digital and depth images captured by an onboard camera. A first distance between the road defect and the vehicle is determined based on the digital and depth images. This first distance is then corrected based on the device distance between the onboard positioning device and the onboard camera to obtain a second distance between the road defect and the vehicle. Finally, the location of the road defect is determined based on the vehicle's position, azimuth angle, and second distance. This method allows the first distance determined based on the digital and depth images to be the distance between the road defect and the onboard camera. The second distance, obtained by correcting the first distance using the distance between the onboard positioning device and the onboard camera, is the distance between the road defect and the onboard positioning device. Since the vehicle's position, azimuth angle, and second distance are all based on the onboard positioning device, determining the location of the road defect based on these factors results in high accuracy.

[0087] Furthermore, by using a TOF camera to capture depth images, real-time measurements can be taken of road defects at a considerable distance. Combined with digital images captured by traffic checkpoint cameras, the distance between road defects and vehicles can be efficiently measured even when the road defects are far away, thereby improving the efficiency of determining the location of road defects.

[0088] In one embodiment, the vehicle-mounted camera includes a first vehicle-mounted camera and a second vehicle-mounted camera. The first vehicle-mounted camera is used to capture digital images, and the second vehicle-mounted camera is used to capture depth images. The above step S220 may specifically include: determining the pixel coordinates of the road defects in the digital image; determining the depth coordinates corresponding to the pixel coordinates in the depth image according to the mapping relationship between the digital image and the depth image; and determining the first distance between the road defects and the vehicle according to the depth value corresponding to the depth coordinates.

[0089] The first vehicle-mounted camera can be a traffic checkpoint camera. The second vehicle-mounted camera can be a TOF camera.

[0090] Pixel coordinates can be the coordinates corresponding to pixels in a digital image, and can be two-dimensional coordinates.

[0091] The depth coordinates can be the coordinates of pixels in the depth image, and can consist of two-dimensional coordinates and a depth value.

[0092] In practice, the controller can identify road defects in digital images, determine the pixel coordinates of the road defects in the digital images, and determine the depth coordinates corresponding to the pixel coordinates of the road defects in the digital images based on the mapping relationship between the digital images and the depth images. The depth value corresponding to the depth coordinates is actually the distance between the road defects and the on-board camera on the vehicle. The first distance between the road defects and the vehicle can be determined based on the depth value.

[0093] For example, according to Figure 3 The pixel coordinates of the center point of the defect area in the digital image are (P3x, P3y). According to the mapping relationship between the digital image and the depth image, x' = x / 2464×1750, y' = y / 2056×480, where x' and y' are the horizontal and vertical coordinates of the pixel in the depth image, respectively, and x and y are the horizontal and vertical coordinates of the pixel in the digital image, respectively. The depth coordinates of the depth image corresponding to the pixel coordinates (P3x, P3y) in the digital image are (P3x / 2464×1750, P3y / 2056×480, D), where D is the depth value, representing the distance between the center point of the defect area on the road and the TOF camera on the vehicle. The first distance between the road defect and the vehicle can be determined based on the depth value D.

[0094] In this embodiment, by determining the pixel coordinates of road defects in the digital image, and based on the mapping relationship between the digital image and the depth image, the depth coordinates corresponding to the pixel coordinates in the depth image are determined. Based on the depth value corresponding to the depth coordinates, the first distance between the road defect and the vehicle is determined. Road defects can be reliably identified based on the digital image. Based on the correspondence between the digital image and the depth image, the distance between the identified road defect and the vehicle camera can be quickly determined, and the distance between the road defect and the vehicle camera can be roughly determined as the distance between the road defect and the vehicle, thus improving the efficiency of determining the location of road defects.

[0095] In one embodiment, the step of determining the first distance between the road defect and the vehicle based on the depth value corresponding to the depth coordinates may specifically include: determining the depth value corresponding to the depth coordinates as the shooting distance for the vehicle-mounted camera to photograph the road defect; determining the projection distance corresponding to the shooting distance based on the shooting distance and the shooting height of the vehicle-mounted camera; and determining the projection distance corresponding to the shooting distance as the first distance between the road defect and the vehicle.

[0096] The shooting distance can be the distance between the vehicle-mounted camera and the road defects.

[0097] The shooting height can be the height of the vehicle-mounted camera relative to the ground.

[0098] In the specific implementation, the controller can determine the depth value corresponding to the depth coordinate as the shooting distance for the vehicle-mounted camera to shoot at road defects. The controller can also obtain the shooting height of the vehicle-mounted camera, calculate the square difference between the shooting distance and the shooting height, and perform a square root operation on the obtained square difference to obtain the projected distance corresponding to the shooting distance. The projected distance is determined as the first distance between the road defect and the vehicle.

[0099] For example, according to Figure 1 The depth value D can be defined as the shooting distance for the vehicle-mounted camera to photograph the damaged areas on the road. By pre-measuring the shooting height H of the vehicle-mounted camera relative to the ground, the projected shooting distance can be... Projection distance can be This is determined to be the first distance.

[0100] In this embodiment, by determining the depth value corresponding to the depth coordinate as the shooting distance for the vehicle-mounted camera to photograph road defects, and by determining the projection distance corresponding to the shooting distance based on the shooting distance and the shooting height of the vehicle-mounted camera, the projection distance corresponding to the shooting distance is determined as the first distance between the road defect and the vehicle. This allows for the rapid determination of the horizontal distance between the road defect and the vehicle-mounted camera, and the rough determination of the horizontal distance as the distance between the road defect and the vehicle, thereby improving the efficiency of determining the location of road defects.

[0101] In one embodiment, step S230 may specifically include: adding the first distance to the equipment distance to obtain the second distance between the road defect and the vehicle.

[0102] In practice, the first distance is actually the horizontal distance between the damaged area on the road and the vehicle-mounted camera, and the device distance is the horizontal distance between the vehicle-mounted camera and the vehicle-mounted positioning device. By adding the first distance and the device distance, the horizontal distance between the damaged area on the road and the vehicle-mounted positioning device can be obtained. The horizontal distance between the damaged area on the road and the vehicle-mounted positioning device is determined as the second distance.

[0103] In this embodiment, by adding the first distance to the device distance, a second distance between the road defect and the vehicle can be obtained. This can correct the first distance and obtain the horizontal distance between the road defect and the vehicle positioning device. Subsequently, the location of the road defect can be determined based on the second distance, which can improve the accuracy of the road defect location determination.

[0104] In one embodiment, the vehicle location includes the vehicle's longitude coordinates and latitude coordinates; step S240 may specifically include: determining the tangent radius corresponding to the vehicle's latitude coordinates, and determining the horizontal displacement of the road defect relative to the vehicle based on the vehicle's azimuth angle and the second distance; determining the longitude change value of the road defect relative to the vehicle based on the horizontal displacement and the tangent radius; and adding the vehicle's longitude coordinates to the longitude change value to obtain the longitude coordinates of the road defect.

[0105] The sectional radius can be the radius of a section of the Earth corresponding to the latitude value.

[0106] The horizontal displacement can be the displacement in the horizontal direction of a spatial rectangular coordinate system.

[0107] In the specific implementation, the vehicle's position can be (longA, latA), where longA is the vehicle's longitude coordinate and latA is the vehicle's latitude coordinate. The radius of the tangent plane corresponding to the vehicle's latitude coordinate latA is ARC*cos(latA), where ARC is the average radius of the equatorial circle. Figure 4 The horizontal displacement d*sinθ of the road defect relative to the vehicle can be obtained from the vehicle's azimuth angle θ and the second distance d. Based on the horizontal displacement d*sinθ and the tangential radius ARC*cos(latA), the longitude change of the road defect relative to the vehicle can be determined as d*sinθ / [ARC*cos(latA)*2π / 360]. Adding the vehicle's longitude coordinate longA to the longitude change value yields the longitude coordinate longC of the road defect. The calculation formula is as follows:

[0108] longC=longA+d*sinθ / [ARC*cos(latA)*2π / 360].

[0109] In this embodiment, by determining the tangent radius corresponding to the vehicle's latitude coordinates, and based on the vehicle's azimuth angle and the second distance, the horizontal displacement of the road defect relative to the vehicle is determined. Based on the horizontal displacement and the tangent radius, the longitude change value of the road defect relative to the vehicle is determined. The longitude coordinates of the vehicle and the longitude change value are added together to obtain the longitude coordinates of the road defect. The location of the road defect can be quickly determined based on the vehicle's position and the distance between the road defect and the vehicle, thus improving the efficiency of determining the location of the road defect.

[0110] In one embodiment, step S240 may further include: determining the vertical displacement of the road defect relative to the vehicle based on the vehicle's azimuth angle and the second distance; determining the latitudinal change value of the road defect relative to the vehicle based on the vertical displacement and a preset average radius of the Earth; and adding the vehicle's latitudinal coordinates to the latitudinal change value to obtain the latitudinal coordinates of the road defect.

[0111] The vertical displacement can be the displacement in the vertical direction of the spatial rectangular coordinate system.

[0112] In specific implementation, the vehicle's position can be (longA, latA), where longA is the vehicle's longitude coordinate and latA is the vehicle's latitude coordinate. Figure 4 The vertical displacement d*cosθ of the road defect relative to the vehicle can be obtained based on the vehicle's azimuth angle θ and the second distance d. The Earth's average radius ARC is preset. Based on the vertical displacement d*cosθ and the Earth's average radius ARC, the latitude change of the road defect relative to the vehicle can be determined as d*cosθ / (ARC*2π / 360). Adding the vehicle's longitude coordinate latA to the latitude change value yields the latitude coordinate latC of the road defect. The calculation formula is as follows:

[0113] latC=latA+d*cosθ / (ARC*2π / 360).

[0114] In this embodiment, the vertical displacement of the road defect relative to the vehicle is determined based on the vehicle's azimuth angle and the second distance. Based on the vertical displacement and the preset average radius of the Earth, the latitude change value of the road defect relative to the vehicle is determined. The latitude coordinates of the vehicle are added to the latitude change value to obtain the latitude coordinates of the road defect. The location of the road defect can be quickly determined based on the vehicle's position and the distance between the road defect and the vehicle, thus improving the efficiency of determining the location of the road defect.

[0115] To facilitate a deeper understanding of the embodiments of this application by those skilled in the art, a specific example will be used for illustration below.

[0116] refer to Figure 1 GPS, TOF cameras, 5-megapixel traffic checkpoint cameras, and edge computing terminals can be installed on vehicles. The GPS, TOF cameras, traffic checkpoint cameras, and edge computing terminals can all use the same time source as NTP (Network Time Protocol) for synchronization.

[0117] The ranging principle of a TOF camera is to continuously send light pulses to the target, then use a sensor to receive the light returning from the object, and obtain the distance to the target by detecting the flight (round trip) time of the light pulses.

[0118] NTP is a protocol used to synchronize computer time. It enables computers to synchronize with their servers or clock sources (such as quartz clocks, GPS, etc.), providing high-precision time correction (less than 1 millisecond difference from the standard on LAN, and tens of milliseconds on WAN), and can prevent malicious attacks through encrypted confirmation.

[0119] The method for determining the location of road defects in this application may specifically include the following steps:

[0120] 1. Configure the GPS installation angle and pole arm value to reduce GPS positioning error; configure dual antennas for GPS, obtain the vehicle's driving azimuth angle through dual antennas, convert the azimuth angle to the NED coordinate system, and reduce GPS module error.

[0121] 2. Manually measure the height H of the TOF / traffic checkpoint camera above the ground.

[0122] 3. Calibrate the distance between the TOF / traffic checkpoint camera and the GPS device. Specifically, a measuring tape can be used to measure the distance L between the TOF / traffic checkpoint camera and the GPS device.

[0123] 4. Calculate the world coordinate system coordinates of the road defect center point, camera, and GPS device. Specifically, pixel coordinate information of the defect area on the road can be obtained through a defect identification algorithm. For example, potholes and cracks can be identified in images captured by traffic checkpoint cameras, and the identified areas can be determined as defect areas. The pixel coordinates corresponding to the defect areas can be obtained, and the pixel coordinates of the center point of the defect area can be calculated in the pixel coordinate system. Assuming the defect area is a rectangle, the coordinates of the upper left corner of the rectangle are (P1x, P1y), the coordinates of the lower right corner of the rectangle are (P2x, P2y), and the formula for calculating the pixel coordinates (P3x, P3y) of the center point of the defect area can be:

[0124] P3x = P1x + (P2x - P1x) / 2,

[0125] P3y = P1y + (P2y - P1y) / 2.

[0126] Depth information is obtained by mapping the pixel coordinates of the defect center to a Time-of-Flight (TOF) camera. Assuming the traffic checkpoint camera has a pixel size of 2464*2056 and the TOF camera has a pixel size of 1750*480, mapping point (P3x, P3y) to a TOF camera pixel yields (P3x / 2464*1750, P3y / 2056*480), from which the TOF pixel can be calculated. The depth information D corresponding to the TOF pixel is then obtained, and the distance from the defect location on the road to the camera is calculated based on depth information D. This allows us to obtain the distance from the location of the disease to the GPS setting.

[0127] 5. Based on the vehicle's azimuth angle θ, the GPS device coordinates (longA, latA), and the distance d from the defect location to the GPS device, the GPS position (longC, latC) of the defect center point on the road can be obtained. The specific formula is as follows:

[0128] longC=longA+d*sinθ / [ARC*cos(latA)*2π / 360],

[0129] latC=latA+d*cosθ / (ARC*2π / 360),

[0130] Where ARC is the average radius of the equatorial circle, approximately 6,371,393 meters.

[0131] The aforementioned method for determining the location of road defects first uses a Time-of-Flight (TOF) camera to measure distance, obtaining the distance from the defect's center point to the camera. This distance is then added to the distance between the camera and the GPS device, improving the GPS accuracy of the defect location. While TOF cameras can measure depth information, their resolution is typically low. By adding traffic checkpoint cameras to acquire image information, and ensuring that both the TOF and traffic checkpoint cameras use the same Field of View (FOV), the accuracy of determining the road defect location can be improved. This facilitates rapid defect location by road maintenance personnel, playing a significant role in defect tracking and maintenance.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] In one embodiment, such as Figure 5 As shown, a road defect location determination system is provided, including a vehicle-mounted positioning device 130, a vehicle-mounted camera 140, and a controller 110; both the vehicle-mounted positioning device 130 and the vehicle-mounted camera 140 are installed on a vehicle;

[0134] The vehicle positioning device 130 is used to measure the vehicle position and vehicle azimuth angle of the vehicle, and send the vehicle position and vehicle azimuth angle to the controller 110;

[0135] The vehicle-mounted camera 140 is used to photograph road defects, obtain digital images and depth images, and send the digital images and depth images to the controller 110;

[0136] The controller 110 is configured to acquire the vehicle position, the vehicle azimuth angle, the digital image, and the depth image; determine a first distance between the road defect and the vehicle based on the digital image and the depth image; correct the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain a second distance between the road defect and the vehicle; and determine the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance.

[0137] In practice, the vehicle-mounted positioning device measures the vehicle's position and azimuth angle, and sends these measurements to the controller. The vehicle-mounted camera can include a first camera and a second camera, which can be installed at the same location on the vehicle and have the same field of view. The first camera captures a digital image and sends it to the controller, while the second camera captures a depth image and sends it to the controller. After acquiring the vehicle position, azimuth angle, digital image, and depth image, the controller can determine the pixel coordinates of road defects in the digital image. Based on the mapping relationship between the digital image and the depth image, it determines the depth coordinates corresponding to the pixel coordinates. The depth coordinates of the road defects on the depth image are obtained. Based on the depth values ​​corresponding to the depth coordinates, the distance between the road defects and the vehicle-mounted camera is obtained. This distance is used as the first distance between the road defects and the vehicle. The device distance between the vehicle positioning device and the vehicle-mounted camera can also be obtained. The first distance is corrected using the device distance to obtain the distance between the road defects and the vehicle positioning device. This distance is used as the second distance between the road defects and the vehicle. Since the vehicle position, vehicle azimuth angle, and second distance are all based on the vehicle positioning device, the location of the road defects can be determined based on the vehicle position, vehicle azimuth angle, and second distance.

[0138] The aforementioned road defect location determination system involves an onboard positioning device measuring the vehicle's position and azimuth angle, and sending these data to a controller. An onboard camera captures images of the road defect, generating digital and depth images, which are also sent to the controller. The controller acquires the vehicle position, azimuth angle, digital images, and depth images. Based on these images, it determines a first distance between the road defect and the vehicle. The first distance is then corrected using the distance between the onboard positioning device and the onboard camera to obtain a second distance between the road defect and the vehicle. Finally, the location of the road defect is determined based on the vehicle position, azimuth angle, and second distance. This system allows the first distance, determined from the digital and depth images, to be used as the distance between the road defect and the onboard camera. Correcting this first distance using the distance between the onboard positioning device and the camera yields a second distance, which is then used as the distance between the road defect and the onboard positioning device. Since the vehicle position, azimuth angle, and second distance are all based on the onboard positioning device, determining the location of the road defect using these parameters results in high accuracy.

[0139] Based on the same inventive concept, this application also provides a road defect location determination device for implementing the road defect location determination method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more road defect location determination device embodiments provided below can be found in the limitations of the road defect location determination method described above, and will not be repeated here.

[0140] In one embodiment, such as Figure 6 As shown, a road defect location determination device is provided, comprising: a parameter acquisition module 410, a distance determination module 420, a distance correction module 430, and a location determination module 440, wherein:

[0141] The parameter acquisition module 410 is used to acquire the vehicle position and azimuth angle of the vehicle measured by the vehicle's on-board positioning device, as well as the digital image and depth image obtained by the vehicle's on-board camera from taking pictures of road defects.

[0142] The distance determination module 420 is used to determine a first distance between the road defect and the vehicle based on the digital image and the depth image;

[0143] The distance correction module 430 is used to correct the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain the second distance between the road defect and the vehicle.

[0144] The location determination module 440 is used to determine the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance.

[0145] The various modules in the aforementioned road defect location determination device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0146] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for determining the location of road defects. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0147] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0148] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0149] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining the location of road defects, characterized in that, The method includes: The vehicle position and azimuth angle measured by the vehicle's onboard positioning device are obtained, as well as the digital and depth images obtained by the vehicle's onboard camera from taking pictures of road defects. Based on the digital image and the depth image, a first distance between the road defect and the vehicle is determined; Based on the device distance between the vehicle positioning device and the vehicle camera, the first distance is corrected to obtain the second distance between the road defect and the vehicle; The location of the road defect is determined based on the vehicle position, the vehicle azimuth angle, and the second distance; The step of determining the first distance between the road defect and the vehicle based on the digital image and the depth image includes: Based on the mapping relationship between the digital image and the depth image, the location of the diseased area in the depth image is determined by the location of the diseased area in the digital image; Obtain depth information corresponding to the location of the diseased area in the depth image, wherein the depth information represents the distance between the diseased area on the road and the vehicle-mounted camera on the vehicle; Determine the initial distance between road defects and vehicles based on depth information; The step of correcting the first distance based on the device distance between the vehicle-mounted positioning device and the vehicle-mounted camera to obtain the second distance between the road defect and the vehicle includes: The first distance is corrected using the device distance to obtain the distance between the road defect area and the vehicle positioning device. The distance between the road defect area and the vehicle positioning device is then used as the second distance between the road defect and the vehicle.

2. The method according to claim 1, characterized in that, The vehicle-mounted camera includes a first vehicle-mounted camera and a second vehicle-mounted camera. The first vehicle-mounted camera is used to capture the digital image, and the second vehicle-mounted camera is used to capture the depth image. Determining the first distance between the road defect and the vehicle based on the digital image and the depth image includes: Determine the pixel coordinates of the road defects in the digital image; Based on the mapping relationship between the digital image and the depth image, determine the depth coordinates in the depth image corresponding to the pixel coordinates; Based on the depth value corresponding to the depth coordinates, a first distance is determined between the road defect and the vehicle.

3. The method according to claim 2, characterized in that, Determining the first distance between the road defect and the vehicle based on the depth value corresponding to the depth coordinates includes: The depth value corresponding to the depth coordinates is determined as the shooting distance for the vehicle-mounted camera to photograph the road defects; Based on the shooting distance and the shooting height of the vehicle-mounted camera, determine the projection distance corresponding to the shooting distance; The projection distance corresponding to the shooting distance is determined as the first distance between the road defect and the vehicle.

4. The method according to claim 1, characterized in that, The step of correcting the first distance based on the device distance between the vehicle-mounted positioning device and the vehicle-mounted camera to obtain the second distance between the road defect and the vehicle includes: The first distance is added to the distance of the device to obtain the second distance between the road defect and the vehicle.

5. The method according to claim 1, characterized in that, The vehicle location includes the vehicle's longitude and latitude coordinates; determining the location of the road defect based on the vehicle location, the vehicle's azimuth angle, and the second distance includes: Determine the tangential radius corresponding to the vehicle's latitude coordinates, and determine the horizontal displacement of the road defect relative to the vehicle based on the vehicle's azimuth angle and the second distance; Based on the horizontal displacement and the tangential radius, determine the longitude variation value of the road defect relative to the vehicle; The longitude coordinates of the vehicle are added to the longitude change value to obtain the longitude coordinates of the road defect.

6. The method according to claim 5, characterized in that, The step of determining the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance further includes: The vertical displacement of the road defect relative to the vehicle is determined based on the vehicle's azimuth angle and the second distance. Based on the vertical displacement and the preset average radius of the Earth, the latitudinal change value of the road defect relative to the vehicle is determined; The latitude coordinates of the vehicle are added to the latitude change value to obtain the latitude coordinates of the road defect.

7. A system for determining the location of road defects, characterized in that, The system includes: a vehicle-mounted positioning device, a vehicle-mounted camera, and a controller; both the vehicle-mounted positioning device and the vehicle-mounted camera are installed on the vehicle. The vehicle positioning device is used to measure the vehicle position and azimuth angle of the vehicle, and send the vehicle position and azimuth angle to the controller; The vehicle-mounted camera is used to photograph road defects, obtain digital images and depth images, and send the digital images and depth images to the controller; The controller is configured to acquire the vehicle position, the vehicle azimuth angle, the digital image, and the depth image; determine a first distance between the road defect and the vehicle based on the digital image and the depth image; correct the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain a second distance between the road defect and the vehicle; and determine the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance. The step of determining the first distance between the road defect and the vehicle based on the digital image and the depth image includes: Based on the mapping relationship between the digital image and the depth image, the location of the diseased area in the depth image is determined by the location of the diseased area in the digital image; Obtain depth information corresponding to the location of the diseased area in the depth image, wherein the depth information represents the distance between the diseased area on the road and the vehicle-mounted camera on the vehicle; Determine the initial distance between road defects and vehicles based on depth information; The step of correcting the first distance based on the device distance between the vehicle-mounted positioning device and the vehicle-mounted camera to obtain the second distance between the road defect and the vehicle includes: The first distance is corrected using the device distance to obtain the distance between the road defect area and the vehicle positioning device. The distance between the road defect area and the vehicle positioning device is then used as the second distance between the road defect and the vehicle.

8. A device for determining the location of road defects, characterized in that, The device includes: The parameter acquisition module is used to acquire the vehicle position and azimuth angle of the vehicle as measured by the vehicle's on-board positioning device, as well as the digital image and depth image obtained by the vehicle's on-board camera from taking pictures of road defects. A distance determination module is used to determine a first distance between the road defect and the vehicle based on the digital image and the depth image; A distance correction module is used to correct the first distance based on the device distance between the vehicle positioning device and the vehicle camera to obtain a second distance between the road defect and the vehicle. The location determination module is used to determine the location of the road defect based on the vehicle position, the vehicle azimuth angle, and the second distance; The distance determination module is further configured to: determine the location of the diseased area in the depth image based on the mapping relationship between the digital image and the depth image; obtain depth information corresponding to the location of the diseased area in the depth image, wherein the depth information represents the distance between the diseased area on the road and the vehicle-mounted camera on the vehicle; and determine a first distance between the road disease and the vehicle based on the depth information. The distance correction module is further configured to: correct the first distance using the device distance to obtain the distance between the road defect area and the vehicle positioning device, and use the distance between the road defect area and the vehicle positioning device as the second distance between the road defect and the vehicle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images