Image processing-based road measurement method and related device

Abstract

Provided in the present disclosure are an image processing-based road measurement method and a related device. The image processing-based road measurement method comprises: on the basis of a preset sampling period, acquiring road images containing road markings, adjacent road images having an overlapping area of a preset size; extracting from the road images marking pixel coordinates of the road markings in a camera coordinate system; on the basis of internal matrix parameters and external matrix parameters of the camera coordinate system, converting the marking pixel coordinates into marking world coordinates in a world coordinate system; on the basis of the overlapping area, converting all of the marking world coordinates into target marking coordinates in the same world coordinate system; and fusing the target marking coordinates to obtain a road measurement curve.

Description

A road measurement method and related equipment based on image processing Technical Field

[0001] This disclosure relates to the field of road surveying, and more particularly to a road surveying method and related equipment based on image processing. Background Technology

[0002] Currently, measuring road turning radii often relies on the vehicle trajectory method from navigation devices. This method uses the vehicle's trajectory to approximate the lane curve, estimating the horizontal curve radius at the turn. For example, a satellite positioning receiver is installed in the vehicle to acquire its satellite coordinates in real time during travel. These coordinates are then smoothly fitted into a curve in chronological order to represent the vehicle's trajectory. Based on this trajectory curve, the curvature can be used to determine the curve segment's position, and the horizontal curve radius can be calculated using the cumulus length method or arc length method, thus obtaining the road turning radius. However, due to satellite signal obstruction in mountainous areas and tunnels, measurement requirements are often not met. Furthermore, measurement accuracy is limited by the vehicle's positioning equipment. Additionally, for multi-lane roads, lane changes can cause the vehicle's trajectory to not accurately reflect changes in the road curve. All these factors contribute to low accuracy and efficiency in road measurement results.

[0003] Summary of the Invention

[0004] This disclosure proposes a road measurement method and related equipment based on image processing, which can, to some extent, solve the technical problems of low accuracy and low efficiency in road measurement results.

[0005] In a first aspect, this disclosure provides a road measurement method based on image processing, comprising:

[0006] Road images, including road markings, are acquired based on a preset sampling period, and adjacent road images include an overlapping area of ​​a preset size.

[0007] Extract the pixel coordinates of the road markings in the camera coordinate system from the road image;

[0008] Based on the internal and external matrix parameters of the camera coordinate system, the pixel coordinates of the tracing line are converted into world coordinates of the tracing line in the world coordinate system.

[0009] Based on the overlapping region, all the world coordinates of the grading lines are converted to target grading coordinates in the same world coordinate system; and

[0010] The road measurement curve is obtained by fusing the coordinates of the target markings.

[0011] A second aspect of this disclosure provides a road measurement device based on image processing, comprising:

[0012] The acquisition module is used to acquire road images including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size;

[0013] The extraction module is used to extract the pixel coordinates of the road markings in the camera coordinate system in the road image;

[0014] The first coordinate transformation module is used to convert the tracing pixel coordinates into tracing world coordinates in the world coordinate system based on the internal matrix parameters and external matrix parameters of the camera coordinate system.

[0015] The second coordinate transformation module is used to convert all the world coordinates of the datum lines into target datum line coordinates in the same world coordinate system based on the overlapping area; and

[0016] The fusion module is used to fuse the coordinates of the target markings to obtain a road measurement curve.

[0017] A third aspect of this disclosure provides an electronic device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs including instructions for performing the method according to the first aspect.

[0018] A fourth aspect of this disclosure provides a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the processors to perform the method described in the first aspect.

[0019] A fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0020] As described above, the road measurement method and related equipment based on image processing disclosed herein capture road images with lane markings at a preset sampling period during vehicle travel, while maintaining sufficient overlap between adjacent road images. The pixel coordinates of the lane markings are automatically extracted using image processing technology, and combined with intrinsic and extrinsic parameters of the image acquisition coordinates, the pixel coordinates of the lane lines are converted into world coordinates. By processing feature points in the overlapping areas of adjacent images, a coordinate transformation relationship is established between images, ultimately unifying the lane marking coordinates in all images to the same world coordinate system, resulting in a complete road measurement curve. This not only improves the accuracy and efficiency of road measurement but also simplifies operation and reduces measurement costs. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a schematic diagram of an image processing-based road measurement architecture according to an embodiment of this disclosure.

[0023] Figure 2 is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of this disclosure.

[0024] Figure 3 is a schematic flowchart of the road measurement method based on image processing according to an embodiment of the present disclosure.

[0025] Figure 4 is a schematic diagram of the target marking coordinates based on image processing according to an embodiment of this disclosure.

[0026] Figure 5 is a schematic diagram of the road measurement curve according to an embodiment of this disclosure.

[0027] Figure 6 is a schematic diagram of an image processing-based road measurement device according to an embodiment of this disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0029] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0030] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0031] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0032] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0033] Figure 1 illustrates a schematic diagram of an image processing-based road measurement architecture according to an embodiment of the present disclosure. Referring to Figure 1, the image processing-based road measurement architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 can be connected via a wired or wireless network 130. The server 110 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, security services, and CDN.

[0034] Terminal 120 can be implemented in hardware or software. For example, when terminal 120 is implemented in hardware, it can be various electronic devices with a display screen and support page display, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal 120 is implemented in software, it can be installed in the electronic devices listed above; it can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module, without specific limitations.

[0035] It should be noted that the image processing-based road measurement method provided in this application embodiment can be executed by either terminal 120 or server 110. It should be understood that the number of terminals, networks, and servers in Figure 1 is merely illustrative and not intended to limit their use. Any number of terminals, networks, and servers can be used depending on implementation needs.

[0036] Figure 2 illustrates a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of this disclosure. As shown in Figure 2, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208, and a bus 210. The processor 202, memory 204, network module 206, and peripheral interface 208 are interconnected within the electronic device 200 via the bus 210.

[0037] Processor 202 may be a Central Processing Unit (CPU), a Neural Processing Unit (NPU), a Microcontroller (MCU), a Programmable Logic Device (DSP), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits. Processor 202 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated as a single logic component. For example, as shown in FIG2, processor 202 may include multiple processors, namely a first processor 202a, a second processor 202b, and a third processor 202c.

[0038] Memory 204 can be configured to store data (e.g., instructions, computer code, etc.). As shown in FIG2, the data stored in memory 204 may include program instructions (e.g., program instructions for implementing the image processing-based road measurement method of the embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 202 can also access the program instructions and data stored in memory 204 and execute the program instructions to operate on the data to be processed. Memory 204 may include volatile storage devices or non-volatile storage devices. In some embodiments, memory 204 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0039] Network module 206 can be configured to provide communication with other external devices to electronic device 200 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above. In some embodiments, network module 206 may include any combination of any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc.

[0040] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0041] Bus 210 can be configured to transfer information between various components of electronic device 200 (e.g., processor 202, memory 204, network module 206, and peripheral interface 208), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0042] It should be noted that although the architecture of the above-described electronic device 200 only shows the processor 202, memory 204, network module 206, peripheral interface 208, and bus 210, in specific implementations, the architecture of the electronic device 200 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the above-described electronic device 200 may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0043] Currently, methods for measuring road turning radii include manual surveying, vehicle trajectory-based methods using navigation equipment, and aerial photogrammetry. The vehicle trajectory method uses the vehicle's trajectory to approximate the lane curve and estimate the horizontal curve radius at the turning point. Specifically, a satellite positioning receiver is installed on the vehicle to collect and store data, acquiring the vehicle's satellite coordinates in real time during travel. These coordinates are then smoothly fitted into a curve in chronological order, which is the vehicle's trajectory. Since the vehicle travels on the road, this curve can be approximated. Based on the trajectory curve, the curve segment position can be determined by curvature, and the horizontal curve radius can be calculated using the ensemble length method or arc length method. However, satellite signals are easily affected by roadside buildings, mountains, tunnels, etc., often making detection impossible or failing to meet measurement requirements, especially in mountainous areas with numerous curves. Combined with inherent errors in satellite positioning information, this leads to poor road detection results. The accuracy of the vehicle trajectory method is limited by the vehicle-mounted positioning equipment, varying from a few centimeters to tens of meters. Using base stations could improve accuracy, but this significantly increases system cost and complexity. For multi-lane roads, vehicle trajectories caused by lane changes do not accurately reflect changes in road curves, and errors are introduced due to the trajectories not properly matching the route direction. Therefore, improving the accuracy and efficiency of road measurement results while reducing measurement costs have become urgent technical problems to be solved.

[0044] In view of this, embodiments of this disclosure provide a road measurement method and related equipment based on image processing. Road images with lane markings are captured at a preset sampling period during vehicle travel, ensuring sufficient overlap between adjacent road images. The pixel coordinates of the lane markings are automatically extracted using image processing technology, and combined with intrinsic and extrinsic parameters of the image acquisition coordinates, the pixel coordinates of the lane markings are converted into world coordinates. By processing feature points in the overlapping areas of adjacent images, coordinate transformation relationships between images are established, ultimately unifying the lane marking coordinates in all images to the same world coordinate system, resulting in a complete road measurement curve. This not only improves the accuracy and efficiency of road measurement but also simplifies operation and reduces measurement costs.

[0045] Referring to Figure 3, Figure 3 shows a schematic flowchart of an image processing-based road measurement method according to an embodiment of the present disclosure. The image processing-based road measurement method according to an embodiment of the present disclosure can be deployed on a server or a terminal. In Figure 3, the image processing-based road measurement method 300 may further include the following steps.

[0046] In step S310, a road image including road markings is acquired based on a preset sampling period, and adjacent road images include an overlapping area of ​​a preset size.

[0047] The preset sampling period refers to image acquisition at pre-defined time or distance intervals. For example, acquiring one image every second or every ten meters. Road markings refer to road markings used to indicate direction, lane boundaries, etc., such as solid or dashed white or yellow lines. Road images refer to road surface images, including road markings, acquired through vehicle-mounted cameras or other image acquisition devices. Overlapping areas refer to the common areas between adjacent images, which can be used for image stitching or coordinate transformation. Preset size refers to the proportion of image dimensions; for example, 50% overlap between two adjacent frames.

[0048] Specifically, data acquisition devices such as cameras, line lasers, and distance sensors can be installed on vehicles. These devices can withstand changes in external lighting conditions; for example, high dynamic range (HDR) cameras can be used to adapt to different light intensities, or infrared cameras can be used to operate at night or in low-light environments. Line lasers and distance sensors help determine the distance between the vehicle and its surroundings, assisting cameras in acquiring more accurate data. A fixed sampling period can be preset according to the needs of the application scenario. For example, if driving on urban roads, one frame per second can be set; if driving on highways, a faster sampling frequency, such as five frames per second, may be needed. The selection of the sampling period needs to consider factors such as vehicle speed, image processing capabilities, and storage capacity. When the vehicle is moving, the cameras installed on the vehicle will periodically capture road images according to the preset sampling period. To ensure sufficient correlation between images, adjacent images can have a certain overlap area. The size of this overlap area is usually a certain proportion of the image width, such as 50%, which ensures the matching degree between images and the consistency of data. For example, a vehicle is traveling on a highway, and its onboard camera is set to capture one frame per second, with an image resolution of 1920x1080 pixels. Since the vehicle's speed is approximately 60 kilometers per hour, or about 16.67 meters per second, the distance traveled per second is relatively small compared to the image width (assuming 1920 pixels). To ensure sufficient overlap between images, the overlap ratio of adjacent images can be set to 50%, meaning each new image covers half the width of the previous image. This ensures good continuity between images even when the vehicle is moving quickly, facilitating subsequent image processing and analysis. Road images acquired in this way can provide fundamental data support for subsequent tasks such as lane detection. Furthermore, because this onboard image acquisition system is unaffected by external lighting conditions, it can continuously and stably acquire high-quality road images regardless of day or night, or even in adverse weather conditions, without the need for additional equipment.

[0049] In step S320, the pixel coordinates of the road markings in the camera coordinate system are extracted from the road image.

[0050] In this context, the camera coordinate system refers to a three-dimensional coordinate system centered on the lens of the image acquisition device, typically used to describe the spatial position of pixels in an image. For example, the coordinate system of an image acquisition device used to capture road images in vehicle-mounted equipment. Road marking pixel coordinates refer to the positional coordinates of road markings in a road image; for instance, they can be represented as two-dimensional coordinate values ​​in the camera coordinate system.

[0051] In some embodiments, extracting the pixel coordinates of the road markings in the camera coordinate system from the road image includes:

[0052] The road image is input into a trained pixel coordinate model to obtain the pixel coordinates of the lane markings; wherein, the pixel coordinate model is trained based on training samples, and the training samples include real road images with pixel coordinates of lane markings.

[0053] The pixel coordinate model, upon receiving a road image, can quickly identify the location of lane markings and output their pixel coordinates. For example, for a given frame, the pixel coordinate model might output the following set of pixel coordinates for lane markings. Specifically, a large number of real road images containing lane markings can be collected as a training dataset. These real road images can cover different road conditions, weather conditions, lighting changes, and other scenarios to ensure the model's generalization ability. For each training image, the location of lane markings can be manually or automatically labeled to generate a labeling file. This labeling file records the pixel coordinates of the lane markings, which can be polygon vertex coordinates or pixel-level masks. For example, for a 1920×1080 pixel image, if the lane marking is a straight line, its edge pixel coordinates can be labeled, such as (500, 400), (600, 450), ...

[0054] Suitable deep learning models for lane marking detection can be selected, including but not limited to: semantic segmentation models such as U-Net and DeepLab, which can output pixel-level labels for the entire image; instance segmentation models such as Mask R-CNN, which can not only distinguish different lane marking instances but also provide accurate pixel-level masks; and edge detection models such as the combination of the Canny edge detection algorithm and deep learning. The selected model is trained using a labeled dataset. During training, the model learns to extract lane marking features from the input image, predicts the position of the lane markings, and outputs the corresponding pixel-level labels.

[0055] Test the trained model to evaluate its performance on unseen data. Based on the test results, adjust model parameters or improve the network architecture to enhance detection accuracy. If the test results are unsatisfactory, further optimize the model by adjusting hyperparameters, increasing training data, etc.

[0056] The trained model can be deployed to real-world applications, such as embedded systems installed in vehicles, to process video streams from onboard cameras in real time and output the pixel coordinates of lane markings.

[0057] In step S330, the calibrator pixel coordinates are converted into calibrator world coordinates in the world coordinate system based on the internal matrix parameters and external matrix parameters of the camera coordinate system.

[0058] The intrinsic matrix parameters (i.e., camera intrinsic parameters) describe the camera's inherent properties, such as focal length and image center point, and can be represented in matrix form. The extrinsic matrix parameters (camera extrinsic parameters) describe the camera's position and orientation relative to the world coordinate system, i.e., rotation and translation parameters, and can be represented as a rotation matrix R and a translation vector T. The datum world coordinates refer to the conversion of datum pixel coordinates into coordinates in the actual world coordinate system using the camera's intrinsic and extrinsic parameters.

[0059] For internal matrix parameters f x It can represent the focal length of the camera in the X direction, f y It can represent the camera's focal length in the Y direction; u0 can represent the x-coordinate of the origin of the image coordinate system in the pixel coordinate system; v0 can represent the y-coordinate of the origin of the image coordinate system in the pixel coordinate system.

[0060] Regarding external matrix parameters, the vehicle's posture changes during movement, leading to significant errors in the statically calibrated rotation and translation matrices. Three parallel laser beams can be used as feature points. Photogrammetry is employed to determine the relationship between the ground plane and the camera coordinate system, dynamically calculating the rotation matrix R' and translation matrix T' for each image. Specifically, three parallel laser emitters can be installed on the vehicle, ensuring their emitted laser beams illuminate the ground, forming distinct feature points. These laser beams should be distributed as wide as possible across the vehicle's width to ensure reliable feature point acquisition across the entire width. As the vehicle moves, the camera captures the feature points formed by the laser beams on the ground. For example, their pixel coordinates in the image might be P1(x1, y1), P2(x2, y2), and P3(x3, y3). These feature points appear as bright spots or lines in the image, and image processing algorithms (such as thresholding and edge detection) can be used to detect their positions in each image. Then, a matching algorithm determines the correspondence between these feature points across different images. Since the laser beams are parallel, the feature points they form on the ground should ideally be on the same horizontal line. The line connecting these three feature points in the image should be roughly parallel to the bottom boundary of the image. By detecting the distribution of these feature points in the image and calculating the angle between the line connecting these three points and the bottom boundary of the image, the tilt angle of the camera relative to the ground can be estimated, and thus the plane on which the ground lies can be estimated. Using the known ground plane information, combined with the positions of the laser feature points, photogrammetry techniques can be used to calculate the camera's rotation and translation matrices. Specifically, using the direction of the line connecting the ground plane and the feature points in the image, the camera's rotation matrix R relative to the ground can be calculated; using the positions of the feature points in the image and the known ground plane information, the camera's translation matrix T relative to the ground can be calculated. Each time a new image is captured, the above steps are repeated to update the rotation and translation matrices. In this way, even if the vehicle's attitude changes during movement, accurate camera extrinsic parameters can be obtained in real time.

[0061] For example, at a certain moment, the camera captures three feature points on the ground formed by a laser beam, with pixel coordinates P1(100, 200), P2(200, 220), and P3(300, 240). Image processing algorithms detect these three feature points and calculate that the line connecting them in the image is approximately parallel to the bottom boundary of the image. Using this information, the camera's rotation matrix relative to the ground can be estimated. The translation matrix is (Unit: meters). This method allows for real-time adjustment of the camera's extrinsic parameters, ensuring that the image data acquired during vehicle movement accurately reflects the surrounding environment.

[0062] In some embodiments, converting the calibrated pixel coordinates into calibrated world coordinates in the world coordinate system based on the internal and external matrix parameters of the camera coordinate system includes:

[0063] Where u is the x-coordinate of the camera coordinate system, v is the y-coordinate of the camera coordinate system, and f x f is the focal length in the X direction of the camera coordinate system. y Let be the focal length in the Y direction of the camera coordinate system, u0 be the x-coordinate of the origin of the camera coordinate system, v0 be the y-coordinate of the origin of the camera coordinate system, and R be the distance from the camera coordinate system to the world coordinate system (x). j y j , z j The rotation matrix of ) is T, and the translation matrix from the camera coordinate system to the world coordinate system is T.

[0064] Specifically, by combining the camera's intrinsic parameters and the dynamically calculated extrinsic parameters, the pixel coordinates of the lane markings can be converted into the world coordinates of the lane markings in the world coordinate system of the image, thereby obtaining the position information of the lane markings in the real world.

[0065] In step S340, all the world coordinates of the datum lines are converted into target datum line coordinates in the same world coordinate system based on the overlapping area.

[0066] Among them, the target marking coordinates can refer to the unification of the marking coordinates in different road images into the same world coordinate system through coordinate transformation, so that subsequent fusion processing can obtain a complete road measurement curve.

[0067] In some embodiments, converting all the world coordinates of the datum lines to target datum line coordinates in the same world coordinate system based on the overlapping region includes:

[0068] A predetermined number of feature points are determined from the overlapping region;

[0069] The intermediate rotation matrix and intermediate translation matrix between adjacent road images are determined based on the world coordinates of the datum lines of the feature points in the adjacent road images.

[0070] Starting with the first frame of the road image, all the world coordinates of the road markings are converted into the same world coordinate system based on the intermediate rotation matrix and the intermediate translation matrix to obtain the corresponding target road marking coordinates.

[0071] To determine the intermediate rotation and translation matrices between adjacent road images based on the datum world coordinates of the feature points in adjacent road images, the following steps can be performed sequentially from the (i+1)th frame road image to the 1st frame road image, where i is a natural number: First, feature points are detected and matched in the overlapping region of the (i+1)th frame road image and the 1st frame road image; then, the datum world coordinates of the feature points in the reference world coordinate system of the (i+1)th frame road image and the datum world coordinates in the reference world coordinate system of the 1st frame road image are determined; finally, the intermediate rotation and translation matrices from the (i+1)th frame road image to the 1st frame road image are determined based on the datum world coordinates of the feature points in the reference world coordinate system of the (i+1)th frame road image and the datum world coordinates in the reference world coordinate system of the 1st frame road image.

[0072] Specifically, feature points can be detected and matched from the overlapping region of two adjacent images. For example, feature point detection algorithms (such as SIFT, SURF, or ORB) can be used to detect feature points from two images, or feature matching algorithms (such as FLANN or BFMatcher) can be used to find corresponding feature points in two images. For example, for each pair of matched feature points P and P' in the i-th and i+1-th road images, their respective world coordinates of the datum line in the reference world coordinate system are d. i 'and d i+1 '. It is possible to calculate from d i+1 'to d i The intermediate rotation matrix R i and the intermediate translation matrix T i That is, solving the system of equations d using the least squares method or other optimization methods. i '=R i ·d i+1 '+T i The intermediate rotation matrix R can be obtained. i and the intermediate translation matrix T i , where d i+1 ' represents the world coordinates of feature point P in the (i+1)th frame of the road image, d i The world coordinates of the feature point P in the i-th frame of the road image.

[0073] In some embodiments, starting with the first frame of the road image, all the world coordinates of the road markings are transformed into the same world coordinate system based on the intermediate rotation matrix and the intermediate translation matrix, and the corresponding target road marking coordinates are obtained. This can be done from the (i+1)th frame of the image to the first frame of the image, where i is a natural number. The following steps are performed sequentially: determining the i-th intermediate road marking coordinate in the world coordinate system of the (i+1)th frame of the road image transformed to the i-th frame of the road image; until i = 1, the target road marking coordinates are obtained. Specifically, this may include:

[0074] The coordinates of the i-th intermediate line in the world coordinate system of the (i+1)-th frame road image after transformation to the i-th frame road image are d. i =R i ·d i+1 +T i , where d i+1 Let R be the world coordinates of the road markings in the (i+1)th frame of the image. i Let T be the intermediate rotation matrix between the (i+1)th frame road image and the ith frame road image. i Let be the intermediate translation matrix between the (i+1)th frame road image and the ith frame road image; i = 0, 1, 2, ..., i is a natural number;

[0075] The coordinates of the intermediate mark in the world coordinate system of the road image in the i-th frame are calculated sequentially until i = 0, and the coordinates of the target mark are obtained.

[0076] This can be achieved by calculating the transformation matrix between adjacent images frame by frame, gradually transforming the lane marking coordinates in each frame to the same reference coordinate system. Specifically, the lane marking coordinates di+1 in the (i+1)th frame are transformed by the rotation matrix Ri and the translation matrix Ti to obtain their intermediate coordinates d in the world coordinate system of the i-th frame. i =R i ·d i+1 +T i This process can start from the last frame and proceed frame by frame until the first frame, ultimately unifying all lane marking coordinates to the world coordinate system of the initial frame image to obtain a complete target marking coordinate sequence, as shown in Figure 4. Figure 4 shows a schematic diagram of target marking coordinates according to an embodiment of this disclosure.

[0077] In step S350, the coordinates of the target markings are fused to obtain the road measurement curve.

[0078] Here, fusion refers to processing multiple road marking coordinate information to eliminate redundancy and errors, resulting in more accurate road marking information. The road measurement curve refers to the actual road shape curve drawn based on the fused target road marking coordinates, which can be used for applications such as navigation and route planning. Through image processing technology, road marking information can be accurately obtained, and through coordinate transformation and fusion techniques, an accurate road shape description can be obtained, as shown in Figure 5. Figure 5 illustrates a schematic diagram of a road measurement curve according to an embodiment of this disclosure.

[0079] In some embodiments, fusing the target marking coordinates to obtain a road measurement curve includes:

[0080] The road measurement curve is obtained by smoothing the target marking coordinates and then fitting the curve.

[0081] Specifically, discrete target line coordinates can be smoothed to remove noise and make the trajectory smoother. Examples include moving average smoothing (smoothing the data by calculating the average within a window), polynomial fitting smoothing (fitting the trajectory using a polynomial function), and interpolation smoothing (smoothing the trajectory using spline interpolation). The smoothed target line coordinates can then be used for curve fitting, for example, polynomial fitting (fitting a curve using a polynomial function).

[0082] In some embodiments, the method 300 further includes:

[0083] The road turning points are determined based on the curved portion of the road measurement curve.

[0084] The curve radius is determined based on the curvature of the curve portion to obtain the turning radius at the road bend.

[0085] A road is composed of many straight and curved segments. The radius of these curved segments, also called the turning radius, affects visibility, speed, and maneuverability, making it a crucial parameter for road design and management. Specifically, for a fitted road measurement curve, the curvature of each curve segment can be calculated: Where x and y are the coordinates of the road measurement curve, x′ and y′ represent the first derivatives of x and y, and x″ and y″ represent the second derivatives of x and y. Further, the turning radius R = 1 / |k| is calculated using the curvature k. The discrete target marking coordinates are smoothed, curve-fitted, and the curvature is calculated to obtain the turning radius at each turn. This can be applied to road design and management, path planning for autonomous vehicles, lane keeping, and other functions.

[0086] As can be seen, the method according to the embodiments of this disclosure, which uses lane marking images captured by an onboard camera and calculates the road turning radius using photogrammetry, can solve the problem of no signal in mountainous areas in existing road measurements. The route coordinates calculated using overlapping lane line images can be accurate to the centimeter, which is sufficient to meet the measurement needs of turning radius and has better repeatability. Lane line images are inherent road markers and can reflect the turning radius better than vehicle travel tracks. The measurement results are not affected by the vehicle trajectory, nor are they limited by terrain, and there are no signal blockages, especially in mountainous sections and tunnel sections; the detection work is not affected by lighting conditions and can be carried out day and night without the need for additional equipment.

[0087] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0088] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Based on the same technical concept, corresponding to any of the above embodiments, this disclosure also provides a road measurement device based on image processing. Referring to Figure 6, the road measurement device based on image processing includes:

[0090] The acquisition module is used to acquire road images including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size;

[0091] The extraction module is used to extract the pixel coordinates of the road markings in the camera coordinate system in the road image;

[0092] The first coordinate transformation module is used to convert the tracing pixel coordinates into tracing world coordinates in the world coordinate system based on the internal matrix parameters and external matrix parameters of the camera coordinate system.

[0093] The second coordinate transformation module is used to convert all the world coordinates of the marking lines into target marking line coordinates in the same world coordinate system based on the overlapping area.

[0094] The fusion module is used to fuse the coordinates of the target markings to obtain a road measurement curve.

[0095] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0096] The apparatus of the above embodiments is used to implement the corresponding image processing-based road measurement method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0097] Based on the same technical concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the image processing-based road measurement method as described in any of the above embodiments.

[0098] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0099] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the image processing-based road measurement method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0100] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0101] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0102] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0103] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A road measurement method based on image processing, comprising: Road images, including road markings, are acquired based on a preset sampling period, and adjacent road images include an overlapping area of ​​a preset size. Extract the pixel coordinates of the road markings in the camera coordinate system from the road image; Based on the internal and external matrix parameters of the camera coordinate system, the pixel coordinates of the tracing line are converted into world coordinates of the tracing line in the world coordinate system. Based on the overlapping area, all the world coordinates of the marking lines are converted into target marking line coordinates in the same world coordinate system; as well as The road measurement curve is obtained by fusing the coordinates of the target markings.

2. The road measurement method based on image processing according to claim 1, wherein, Extracting the pixel coordinates of the road markings in the camera coordinate system from the road image includes: The road image is input into a trained pixel coordinate model to obtain the pixel coordinates of the lane markings; wherein, the pixel coordinate model is trained based on training samples, and the training samples include real road images with pixel coordinates of lane markings.

3. The road measurement method based on image processing according to claim 1, wherein, Based on the overlapping region, converting all the world coordinates of the datum lines to target datum line coordinates in the same world coordinate system includes: A predetermined number of feature points are determined from the overlapping region; The intermediate rotation matrix and intermediate translation matrix between adjacent road images are determined based on the world coordinates of the datum lines of the feature points in the adjacent road images. Starting with the first frame of the road image, all the world coordinates of the road markings are converted into the same world coordinate system based on the intermediate rotation matrix and the intermediate translation matrix to obtain the corresponding target road marking coordinates.

4. The road measurement method based on image processing according to claim 3, wherein, Starting with the first frame of the road image, based on the intermediate rotation matrix and the intermediate translation matrix, all the world coordinates of the road markings are transformed into the same world coordinate system to obtain the corresponding target road marking coordinates, including: From the (i+1)th frame to the first frame, where i is a natural number, execute the following sequentially: Determine the coordinates of the i-th intermediate marker line in the world coordinate system of the (i+1)-th frame road image after transforming it to the i-th frame road image; Until i = 1, the coordinates of the target marking line are obtained.

5. The road measurement method based on image processing according to claim 4, wherein, Determining the coordinates of the i-th intermediate marker in the world coordinate system of the (i+1)-th frame road image transformed to the i-th frame road image includes: based on expression d i =R i ·d i+1 +T i Determine the coordinates of the i-th intermediate datum line in the world coordinate system of the i-th frame of the road image; where d i+1 R represents the world coordinates of the road markings in the (i+1)th frame of the image; i T is the intermediate rotation matrix between the (i+1)th frame road image and the ith frame road image; i It is the intermediate translation matrix between the road image in frame (i+1) and the road image in frame (i).

6. The road measurement method based on image processing according to claim 1, wherein, Converting the calibrated pixel coordinates to calibrated world coordinates in the world coordinate system based on the internal and external matrix parameters of the camera coordinate system includes: Where u is the x-coordinate of the tagged line pixel coordinates; v is the y-coordinate of the tagged line pixel coordinates; f x f is the focal length in the X direction of the camera coordinate system; y Let be the focal length in the Y direction of the camera coordinate system; u0 be the x-coordinate of the origin of the camera coordinate system; v0 be the y-coordinate of the origin of the camera coordinate system; R be the rotation matrix from the camera coordinate system to the world coordinate system; T be the translation matrix from the camera coordinate system to the world coordinate system; (x j y j , z j () represents the world coordinates of the calibrator.

7. The road measurement method based on image processing according to claim 3, wherein, Determining the intermediate rotation matrix and intermediate translation matrix between adjacent road images based on the world coordinates of the datum lines of the feature points in adjacent road images includes: From the (i+1)th frame of the road image to the first frame of the road image, where i is a natural number, execute the following sequentially: Detect and match feature points in the overlapping region of the (i+1)th frame road image and the i-th frame road image; Determine the datum world coordinates of the feature point in the reference world coordinate system of the road image in frame (i+1) and the datum world coordinates in the reference world coordinate system of the road image in frame i; and Based on the datum world coordinates of the feature points in the reference world coordinate system of the (i+1)th frame road image and the datum world coordinates in the reference world coordinate system of the i-th frame road image, determine the intermediate rotation matrix and intermediate translation matrix from the (i+1)th frame road image to the i-th frame road image.

8. The road measurement method based on image processing according to claim 7, wherein, Determining the intermediate rotation and translation matrices from the (i+1)th frame road image to the i-th frame road image based on the datum world coordinates of the feature points in the reference world coordinate system of the (i+1)th frame road image and the datum world coordinates in the reference world coordinate system of the i-th frame road image includes: solving the system of equations d i '=R i ·d i+1 '+T i The intermediate rotation matrix R is obtained. i and the intermediate translation matrix T i ; where d i+1 ' represents the world coordinates of feature point P in the (i+1)th frame of the road image; d i ' represents the world coordinates of feature point P' in the i-th frame of the road image.

9. The road measurement method based on image processing according to claim 1, wherein, The process of fusing the target marking coordinates to obtain the road measurement curve includes: smoothing the target marking coordinates and then fitting the curve to obtain the road measurement curve.

10. The road measurement method based on image processing according to claim 1, further comprising: The road turning points are determined based on the curved portion of the road measurement curve. as well as The curve radius is determined based on the curvature of the curve portion to obtain the turning radius at the road bend.

11. A road measurement device based on image processing, comprising: The acquisition module is used to acquire road images including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size; The extraction module is used to extract the pixel coordinates of the road markings in the camera coordinate system in the road image; The first coordinate transformation module is used to convert the tracing pixel coordinates into tracing world coordinates in the world coordinate system based on the internal matrix parameters and external matrix parameters of the camera coordinate system. The second coordinate transformation module is used to convert all the world coordinates of the marking lines into target marking line coordinates in the same world coordinate system based on the overlapping area. as well as The fusion module is used to fuse the coordinates of the target markings to obtain a road measurement curve.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 10.

Images