A road measurement method based on image processing and related equipment

Abstract

The present disclosure provides a road measurement method based on image processing and related equipment. The method comprises: acquiring a road image including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size; extracting the marking pixel coordinates of the road markings in the road image in a camera coordinate system; converting the marking pixel coordinates into marking world coordinates in a world coordinate system based on internal matrix parameters and external matrix parameters of the camera coordinate system; converting all the marking world coordinates into target marking coordinates in the same world coordinate system based on the overlapping area; and fusing the target marking coordinates to obtain a road measurement curve.

Description

Technical Field

[0001] The present disclosure relates to the field of road measurement, and in particular to a road measurement method based on image processing and related equipment. Background Art

[0002] At present, the measurement of road turning radius is often based on the driving trajectory method of navigation equipment, that is, the vehicle driving trajectory is used instead of the lane curve to estimate the flat curve radius of the road at the turn. For example, a satellite positioning receiver is installed on the vehicle to obtain the satellite positioning coordinates of the vehicle's location in real time during the vehicle's driving process; these coordinates are smoothly fitted into a curve in chronological order as the vehicle's driving trajectory. According to the trajectory curve, the position of the curve segment can be determined by the curvature, and the flat curve radius can be calculated by the Xuanchang method or the arc length method to obtain the road turning radius. However, due to the obstruction of satellite signals in mountainous areas and tunnels, the measurement requirements are often not met, and the measurement accuracy is limited by the on-board positioning equipment. In addition, for multi-lane roads, the driving trajectory caused by the vehicle changing lanes cannot truly reflect the changes in the road curve. These all lead to low accuracy and low measurement efficiency in road measurement results. Summary of the invention

[0003] The present disclosure proposes a road measurement method and related equipment based on image processing, so as to solve the technical problems of low accuracy and low measurement efficiency of road measurement results to a certain extent.

[0004] In a first aspect of the present disclosure, a road measurement method based on image processing is provided, comprising:

[0005] Acquire a road image including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size;

[0006] Extracting the pixel coordinates of the road markings in the road image in the camera coordinate system;

[0007] The pixel coordinates of the reticle are converted into the world coordinates of the reticle in the world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system;

[0008] Based on the overlapping area, all the reticle world coordinates are converted into target reticle coordinates in the same world coordinate system;

[0009] The target marking coordinates are fused to obtain a road measurement curve.

[0010] In a second aspect of the present disclosure, a road measurement device based on image processing is provided, comprising:

[0011] An acquisition module, configured to acquire a road image including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size;

[0012] An extraction module, used for extracting the pixel coordinates of the road markings in the road image in the camera coordinate system;

[0013] A first coordinate conversion module, configured to convert the pixel coordinates of the marking line into the world coordinates of the marking line in a world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system;

[0014] A second coordinate conversion module, used for converting all the reticle world coordinates into target reticle coordinates in the same world coordinate system based on the overlapping area;

[0015] The fusion module is used to fuse the target marking coordinates to obtain a road measurement curve.

[0016] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0017] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0018] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0019] From the above, it can be seen that the present disclosure provides a road measurement method and related equipment based on image processing, which captures road images with lane markings according to a preset sampling period during vehicle driving, and maintains sufficient overlapping areas between adjacent road images. The pixel coordinates of the lane markings are automatically extracted based on image processing technology, and the pixel coordinates of the lane markings are converted into world coordinates in combination with the internal and external parameters of the image acquisition coordinates. By processing the feature points of the overlapping areas of adjacent images, a coordinate conversion relationship between images is established, and finally the lane marking coordinates in all images are unified into the same world coordinate system to obtain a complete road measurement curve. It can not only improve the accuracy and efficiency of road measurement, but also simplify the operation and reduce the measurement cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 Schematic diagram of a road measurement architecture based on image processing according to an embodiment of the present disclosure.

[0022] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0023] Figure 3 The figure is a flowchart of a road measurement method based on image processing according to an embodiment of the present disclosure.

[0024] Figure 4 Schematic diagram of target reticle coordinates based on image processing according to an embodiment of the present disclosure.

[0025] Figure 5 Schematic diagram of a road measurement curve according to an embodiment of the present disclosure.

[0026] Figure 6 Schematic diagram of a road measurement device based on image processing according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0028] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0029] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0031] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0032] Figure 1 FIG. 1 is a schematic diagram showing a road measurement architecture based on image processing according to an embodiment of the present disclosure. Figure 1 The image processing-based road measurement architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0033] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0034] It should be noted that the road measurement method based on image processing provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0035] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0036] Processor 202 may be a central processing unit (CPU), a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, the processor 202 may include a plurality of processors, namely a first processor 202a, a second processor 202b, and a third processor 202c.

[0037] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (for example, program instructions for implementing the road measurement method based on image processing of the embodiment of the present disclosure) and data to be processed (for example, the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0038] The network module 206 can be configured to provide the electronic device 200 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 206 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0039] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0040] The bus 210 may be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0041] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0042] At present, there are two methods for measuring the turning radius of roads: manual surveying and mapping, navigation equipment-based driving trajectory method, and aerial photogrammetry method. The driving trajectory method uses the vehicle's driving trajectory instead of the lane curve to estimate the radius of the flat curve of the road at the turn. Specifically, a satellite positioning receiver acquisition storage device is installed on the vehicle to obtain the satellite positioning coordinates of the vehicle's location in real time during the vehicle's driving. 2. Smoothly fitting these coordinates into a curve in chronological order is the vehicle's driving trajectory. Because the vehicle is driving on the road, it can be approximated as a road curve. According to the trajectory curve, the position of the curve segment can be determined by the curvature, and then the radius of the flat curve can be calculated by the Xuanchang method or the arc length method. However, since satellite signals are easily affected by roadside buildings, peaks, tunnels, etc., the signal is often unable to be detected when it is blocked, or often fails to meet the measurement requirements, especially in mountainous areas where there are relatively many bends, coupled with the measurement errors caused by the errors in the satellite positioning information itself, resulting in poor road detection results. The accuracy of the driving trajectory method is limited by the vehicle-mounted positioning equipment and the accuracy is not high, ranging from a few centimeters to more than ten meters. If the base station method is used, the accuracy can be improved, but the system cost and system complexity are greatly increased. For multi-lane roads, the driving trajectory caused by the vehicle changing lanes cannot truly reflect the changes in the road curve, and the driving trajectory cannot match the direction of the route well, which introduces errors. Therefore, how to improve the accuracy and efficiency of road measurement results and reduce measurement costs have become technical problems that need to be solved urgently.

[0043] In view of this, the embodiments of the present disclosure provide a road measurement method and related equipment based on image processing. By capturing road images with lane markings according to a preset sampling period during vehicle driving, and maintaining sufficient overlapping areas between adjacent road images. Automatically extract the pixel coordinates of the lane markings based on image processing technology, and convert the pixel coordinates of the lane markings into world coordinates in combination with the intrinsic and extrinsic parameters of the image acquisition coordinates. By processing the feature points of the overlapping areas of adjacent images, a coordinate conversion relationship between images is established, and finally the lane marking coordinates in all images are unified into the same world coordinate system to obtain a complete road measurement curve. It can not only improve the accuracy and efficiency of road measurement, but also simplify the operation and reduce the measurement cost.

[0044] See also Figure 3 , Figure 3 A schematic flow chart of a road measurement method based on image processing according to an embodiment of the present disclosure is shown. The road measurement method based on image processing according to an embodiment of the present disclosure can be deployed on a server or a terminal. Figure 3 In the embodiment, the road measurement method 300 based on image processing may further include the following steps.

[0045] In step S310, a road image including road markings is acquired based on a preset sampling period, and adjacent road images include an overlapping area of ​​a preset size.

[0046] Among them, the preset sampling period may refer to image acquisition at a preset time interval or distance interval. For example, an image is acquired every one second or every ten meters. Road markings may refer to marking lines on the road used to indicate directions, lane boundaries and other information, such as white or yellow solid lines, dotted lines, etc. A road image may refer to an image of the road surface including road markings obtained by a vehicle-mounted camera or other image acquisition device. An overlapping area may refer to a common area between adjacent images, which can be used for image stitching or coordinate transformation. A preset size may refer to the ratio of the image size, for example, there is a 50% overlap between two adjacent frames of images.

[0047] Specifically, data acquisition devices such as cameras, line lasers, and distance sensors can be installed on the vehicle. These data acquisition devices can withstand changes in external lighting conditions, such as using high dynamic range (HDR) cameras to adapt to different light intensities, or using infrared cameras to work at night or in low light environments. Line lasers and distance sensors can help determine the distance between the vehicle and the surrounding environment, and assist the camera in obtaining more accurate data. A fixed sampling period can be preset according to the needs of the application scenario. For example, if driving on a city road, it can be set to collect one frame of image per second; if it is a highway, a faster acquisition frequency may be required, such as collecting five frames of image per second. The selection of the sampling period needs to consider factors such as vehicle speed, image processing capabilities, and storage capacity. When the vehicle is driving, the camera installed on the vehicle will regularly capture road images according to the preset sampling period. In order to ensure that there is sufficient correlation between the images, two adjacent images can have a certain overlap area. The size of this overlap area is usually a certain proportion of the image width, such as 50%, so that the matching degree between images and the consistency of data can be guaranteed. For example, a vehicle is driving on a highway, and the on-board camera installed on it is set to collect one frame of image per second, with an image resolution of 1920x1080 pixels. Since the vehicle speed is about 60 kilometers per hour, that is, about 16.67 meters per second, the distance traveled per second is quite small relative to the image width (assuming 1920 pixels). In order to ensure that there is enough overlap between images, the overlap rate of adjacent images can be set to 50%, that is, each new image covers half the width of the previous image. In this way, even if the vehicle moves quickly, the images can still maintain good continuity, which is convenient for subsequent image processing and analysis. The road images collected in this way can provide basic data support for subsequent tasks such as lane detection. In addition, since the on-board image acquisition system is not affected by external lighting conditions, it can continuously and stably obtain high-quality road images without additional equipment, whether during the day or at night, or even in bad weather conditions.

[0048] In step S320, the pixel coordinates of the road markings in the road image in the camera coordinate system are extracted.

[0049] The camera coordinate system may refer to a three-dimensional coordinate system centered on the lens of the image acquisition device, which is usually used to describe the spatial position of pixels in an image. For example, the coordinate system of an image acquisition device used to acquire road images in a vehicle-mounted device. The marking pixel coordinates may refer to the position coordinates of road markings in a road image, which may be expressed as two-dimensional coordinate values ​​in the camera coordinate system, for example.

[0050] In some embodiments, extracting the pixel coordinates of the road markings in the road image in the camera coordinate system includes:

[0051] The road image is input into a trained pixel coordinate model to obtain the lane marking pixel coordinates; wherein the pixel coordinate model is trained based on training samples, and the training samples include real road images annotated with lane marking pixel coordinates.

[0052] Among them, after receiving the road image, the pixel coordinate model can quickly identify the position of the lane marking and output the pixel coordinates of the lane marking. For example, for a certain frame of image, the pixel coordinate model may output the following set of pixel coordinates of lane markings. Specifically, a large number of real road images containing lane markings can be collected as training data sets. These real road images can cover different road conditions, weather conditions, lighting changes and other scenes to ensure the generalization ability of the model. For each training image, the position of the lane marking can be manually or automatically annotated to generate an annotation file. The annotation file records the pixel coordinates of the lane marking, which can be polygon vertex coordinates or pixel-level masks. For example, for a 1920×1080 pixel image, if the lane marking is a straight line, the pixel coordinates of its edge can be annotated, such as (500, 400), (600, 450),...

[0053] You can choose a deep learning model suitable for lane marking detection, including but not limited to: semantic segmentation models: such as U-Net, DeepLab, etc., which can output pixel-level labels for the entire image; instance segmentation models: such as Mask R-CNN, which can not only distinguish different lane marking instances, but also provide accurate pixel-level masks; edge detection models: such as the combination of Canny edge detection algorithm and deep learning. Use the labeled dataset to train the selected model. During the training process, the model will learn to extract the features of lane markings from the input image, predict the location of lane markings, and output the corresponding pixel-level labels.

[0054] Test the trained model to evaluate its performance on unseen data. Based on the test results, adjust the model parameters or improve the network architecture to improve detection accuracy. If the test results are not satisfactory, you can further optimize the model by adjusting hyperparameters, increasing training data, etc.

[0055] Deploy the trained model to a real-world application, such as an embedded system installed in a vehicle, to process the video stream from the on-board camera in real time and output the pixel coordinates of the lane markings.

[0056] In step S330, the reticle pixel coordinates are converted into reticle world coordinates in a world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system.

[0057] Among them, the internal matrix parameters (i.e., camera intrinsic parameters) can be used to describe the properties of the camera itself, such as focal length, image center point, etc., and can be expressed in matrix form. The external matrix parameters (camera extrinsic parameters) can be used to describe the position and posture of the camera relative to the world coordinate system, that is, the rotation parameters and translation parameters, which can be expressed as the rotation matrix 𝑅 and the translation vector T. The world coordinates of the reticle can refer to the conversion of the reticle pixel coordinates into the coordinate values ​​in the actual world coordinate system through the internal and external parameters of the camera.

[0058] For the internal matrix parameters , It can represent the focal length of the camera in the X direction. It can represent the focal length of the camera in the Y direction; It can represent the horizontal coordinate of the origin of the image coordinate system in the pixel coordinate system. It can represent the ordinate of the origin of the image coordinate system in the pixel coordinate system.

[0059] For external matrix parameters, the posture of the vehicle will change during driving, and the static calibration rotation matrix and translation matrix errors are relatively large. You can use three parallel laser beams as feature points, use photogrammetry technology to determine the relationship between the ground plane and the camera coordinate system, and dynamically calculate the rotation matrix of each photo. R’ and translation matrix T’ Specifically, three parallel laser emitters can be installed on the vehicle to ensure that the laser beams they emit can illuminate the ground to form obvious feature points. These laser beams should be distributed as much as possible within the width of the vehicle to obtain reliable feature points across the entire width of the vehicle. When the vehicle is driving, the camera will capture the feature points formed by the laser beams on the ground. For example, the pixel coordinates in the image are P 1 ( x 1 , y 1 ), P2 ( x 2 , y 2 ), P 3 ( x 3 , y 3 ). These feature points appear as bright spots or lines in the image, and image processing algorithms (such as threshold segmentation, edge detection, etc.) can be used to detect the positions of these laser feature points from each image. Then, the corresponding relationship between these feature points in different images is determined by a matching algorithm. Since the laser beams are parallel, the feature points they form on the ground should ideally be on the same horizontal line, and the line connecting the three feature points in the image should be roughly parallel to the bottom boundary of the image. By detecting the distribution of these feature points in the image and calculating the angle between the line connecting the three points in the image and the bottom boundary of the image, the tilt angle of the camera relative to the ground can be estimated, and then the plane of the ground can be estimated. Using the known ground plane information and the position of the laser feature points, photogrammetry technology can be used to calculate the rotation matrix and translation matrix of the camera. Among them, the rotation matrix of the camera relative to the ground can be calculated using the direction of the ground plane and the line connecting the feature points in the image. R ; Through the position of the feature points in the image and the known ground plane information, the translation matrix of the camera relative to the ground can be calculated T Each time a new image is captured, the above steps are repeated to update the rotation matrix and translation matrix. In this way, even if the vehicle's posture changes during driving, accurate camera extrinsics can be obtained in real time.

[0060] For example, at a certain moment, the camera captures three feature points formed by the laser beam on the ground, and their pixel coordinates are: P 1 (100, 200), P 2 (200, 220), P 3 (300, 240), these three feature points are detected by image processing algorithms, and their connecting line in the image is calculated to be roughly parallel to the bottom boundary of the image. Using this information, the rotation matrix of the camera relative to the ground can be estimated as , the translation matrix is (Unit: meter). In this way, the external parameters of the camera can be adjusted in real time to ensure that the image data acquired during the vehicle's driving process can accurately reflect the environmental information around the vehicle.

[0061] In some embodiments, converting the reticle pixel coordinates into reticle world coordinates in a world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system includes:

[0062] ,

[0063] in, is the horizontal coordinate of the camera coordinate system, v is the ordinate of the camera coordinate system, is the focal length in the X direction of the camera coordinate system, is the Y-direction focal length of the camera coordinate system, is the horizontal coordinate of the origin of the camera coordinate system, is the ordinate of the origin of the camera coordinate system, R From the camera coordinate system to the world coordinate system ( x j , y j , z j ), T The translation matrix from the camera coordinate system to the world coordinate system.

[0064] Specifically, by combining the camera's intrinsic parameters and the dynamically calculated extrinsic parameters, the pixel coordinates of the lane marking can be converted into the world coordinates of the lane marking in the world coordinate system where the image is located, thereby obtaining the position information of the lane marking in the real world.

[0065] In step S340, all the reticle world coordinates are converted into target reticle coordinates in the same world coordinate system based on the overlapping area.

[0066] The target marking line coordinates may refer to unifying the marking line coordinates in different road images into the same world coordinate system through coordinate transformation, so as to obtain a complete road measurement curve through subsequent fusion processing.

[0067] In some embodiments, converting all the reticle world coordinates into target reticle coordinates in the same world coordinate system based on the overlapping area includes:

[0068] Determining a preset number of feature points from the overlapping area;

[0069] Determining an intermediate rotation matrix and an intermediate translation matrix between adjacent road images based on the marking line world coordinates of the feature points in adjacent road images;

[0070] Taking the first frame of the road image as the starting point, all the marking line world coordinates are converted to the same world coordinate system based on the intermediate rotation matrix and the intermediate translation matrix to obtain the corresponding target marking line coordinates.

[0071] Specifically, feature points can be detected and matched from the overlapping areas of two adjacent images. For example, feature point detection algorithms (such as SIFT, SURF, or ORB, etc.) can be used to detect feature points from two images, or feature matching algorithms (such as FLANN or BFMatcher, etc.) can be used to find corresponding feature points in two images. For example, for each pair of matching feature points P and P' in the i-th frame road image and the i+1-th frame road image, their respective marking world coordinates in the reference world coordinate system are d i ’ and d i+1 ’ . It can be calculated from d i+1 ’ arrive d i ’ The intermediate rotation matrix R i and the intermediate translation matrix T i That is, solving the system of equations by the least squares method or other optimization methods d i ’ = R i · d i+1 ’ + T i , we can get the intermediate rotation matrix R i and the intermediate translation matrix T i ， in, d i+1 ’ For the i+1 The world coordinates of the feature point P in the frame road image, d i ’ No. i The world coordinates of the feature point P' in the frame road image.

[0072] In some embodiments, starting from the first frame of the road image, all the marking world coordinates are converted into the same world coordinate system based on the intermediate rotation matrix and the intermediate translation matrix to obtain the corresponding target marking coordinates, including:

[0073] The coordinates of the i-th middle marking line in the world coordinate system when the i+1-th frame road image is converted to the i-th frame road image are: d i = Ri · d i+1 + T i ,in, d i+1 For the i+1 The world coordinates of the markings in the frame road image, R i is the intermediate rotation matrix between the i+1th frame road image and the ith frame road image, T i is the intermediate translation matrix between the i+1th frame road image and the ith frame road image; i=0, 1, 2, ..., i is a natural number;

[0074] The i-1th intermediate marking line coordinates in the world coordinate system of the i-1th frame of road image converted to the i-1th frame of road image are calculated sequentially until i=0, and the target marking line coordinates are obtained.

[0075] Among them, the lane marking coordinates in each frame of the image can be gradually converted to the same reference coordinate system by calculating the transformation matrix between adjacent images frame by frame. Specifically, i +1 Lane marking coordinates in the image frame d +1 after rotation matrix Ri and translation matrix Ti The transformation of i The intermediate coordinates of the frame image in the world coordinate system d i = R i · d i+1 + T i This process can start from the last frame image, and calculate frame by frame until the first frame image, and finally unify the coordinates of all lane markings into the world coordinate system of the initial frame image to obtain a complete target marking coordinate sequence, such as Figure 4 As shown, Figure 4 A schematic diagram showing target reticle coordinates according to an embodiment of the present disclosure is shown.

[0076] In step S350, the target marking line coordinates are fused to obtain a road measurement curve.

[0077] Fusion can refer to data processing of multiple marking coordinate information to eliminate redundancy and errors and obtain more accurate road marking information. Road measurement curves can refer to actual road shape curves drawn based on the fused target marking coordinates, which can be used for navigation, path planning and other applications. Through image processing technology, road marking information can be accurately obtained, and through coordinate conversion and fusion technology, accurate road shape description can be obtained, such as Figure 5As shown, Figure 5 A schematic diagram of a road measurement curve according to an embodiment of the present disclosure is shown.

[0078] In some embodiments, fusing the target marking line coordinates to obtain a road measurement curve includes:

[0079] The target marking line coordinates are smoothed and then fitted to obtain the road measurement curve.

[0080] Specifically, the discrete target reticle coordinates can be smoothed to remove noise and make the trajectory smoother. For example, moving average smoothing: smoothing the data by calculating the average value within the window; polynomial fitting smoothing: fitting the trajectory using a polynomial function; interpolation smoothing: using a spline interpolation method to smooth the trajectory. The smoothed target reticle coordinates are used for curve fitting, for example, polynomial fitting: fitting the curve using a polynomial function.

[0081] In some embodiments, the method 300 further includes:

[0082] determining a road bend based on a curve portion in the road measurement curve;

[0083] A curve radius is determined based on the curvature of the curve portion to obtain a turning radius at a curve of the road.

[0084] The road is composed of many straight and curved segments. The radius of these circular curved segments is also called the turning radius. The turning radius affects the driving line of sight, driving speed and vehicle passability, and is an important parameter in road design and management. Specifically, for the fitted road measurement curve, the curvature of each curved segment can be calculated: k = ,in, x and y are the coordinates of the road measurement curve, x 'and y 'express x and y The first derivative of x ''and y ''express x and y The second-order derivative of . Further, the curvature k is used to calculate the turning radius R=1 / |k|. The discrete target marking coordinates are smoothed, curve fitted, and curvature calculated to finally obtain the turning radius of each turn. It can be applied to road design and management, path planning of autonomous vehicles, lane keeping, and other functions.

[0085] It can be seen that according to the method of the embodiment of the present disclosure, the lane marking image taken by the vehicle-mounted camera is used to calculate the road turning radius using photogrammetry technology, which can solve the problem of no signal in mountainous areas in existing road measurements. The route coordinates calculated using overlapping lane line images can be accurate to centimeters, which is sufficient to meet the measurement needs of the turning radius and has better repeatability. The lane line image is an inherent marker of the road and can better reflect the turning radius than the vehicle's running track. The measurement result is not affected by the driving trajectory, nor is it restricted by the terrain. There is no situation where the signal is blocked, especially mountainous sections and tunnel sections can also be detected; the detection work is not affected by light, and detection can be performed day and night without the need for additional equipment.

[0086] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0087] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a road measurement device based on image processing, see Figure 6 , the road measurement device based on image processing, the device comprising:

[0089] An acquisition module, configured to acquire a road image including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size;

[0090] An extraction module, used for extracting the pixel coordinates of the road markings in the road image in the camera coordinate system;

[0091] A first coordinate conversion module, configured to convert the pixel coordinates of the marking line into the world coordinates of the marking line in a world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system;

[0092] A second coordinate conversion module, used for converting all the reticle world coordinates into target reticle coordinates in the same world coordinate system based on the overlapping area;

[0093] The fusion module is used to fuse the target marking coordinates to obtain a road measurement curve.

[0094] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0095] The device of the above embodiment is used to implement the corresponding road measurement method based on image processing in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0096] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the road measurement method based on image processing as described in any of the above embodiments.

[0097] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0098] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the road measurement method based on image processing as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0099] It should be understood by those skilled in the art that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0100] In addition, to simplify the description and discussion, and in order not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0101] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0102] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A road measurement method based on image processing, comprising: Acquire a road image including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size; Extracting the pixel coordinates of the road markings in the road image in the camera coordinate system; The pixel coordinates of the reticle are converted into the world coordinates of the reticle in the world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system; Converting all the reticle world coordinates into target reticle coordinates in the same world coordinate system based on the overlapping area includes: Determining a preset number of feature points from the overlapping area; Determining an intermediate rotation matrix and an intermediate translation matrix between adjacent road images based on the marking line world coordinates of the feature points in adjacent road images; Taking the first frame of road image as the starting point, based on the intermediate rotation matrix and the intermediate translation matrix, all the marking world coordinates are converted to the same world coordinate system to obtain the corresponding target marking coordinates, including: the i+1 frame of road image is converted to the i-th frame of road image in the world coordinate system of the i-th intermediate marking coordinate is d i = R i · d i+1 + T i ,in, d i+1 is the world coordinate of the marking line of the i+1th frame road image, R i is the intermediate rotation matrix between the i+1th frame road image and the ith frame road image, T i is the intermediate translation matrix between the i+1th frame road image and the ith frame road image; i=0, 1, 2, ..., i is a natural number; sequentially calculate the i-1th intermediate marking line coordinates in the world coordinate system of the i-1th frame road image converted to the i+1th frame road image until i=0, and obtain the target marking line coordinates; The target marking coordinates are fused to obtain a road measurement curve.

2. The road measurement method based on image processing according to claim 1, wherein: Extracting the pixel coordinates of the road markings in the road image in the camera coordinate system includes: The road image is input into a trained pixel coordinate model to obtain the lane marking pixel coordinates; wherein the pixel coordinate model is trained based on training samples, and the training samples include real road images annotated with lane marking pixel coordinates.

3. The road measurement method based on image processing according to claim 1, wherein: The pixel coordinates of the reticle are converted into world coordinates of the reticle in a world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system, including: ， in, is the horizontal coordinate of the pixel coordinate of the marking line, v is the vertical coordinate of the pixel coordinate of the marking line, is the focal length in the X direction of the camera coordinate system, is the Y-direction focal length of the camera coordinate system, is the horizontal coordinate of the origin of the camera coordinate system, is the ordinate of the origin of the camera coordinate system, R is the rotation matrix from the camera coordinate system to the world coordinate system, T is the translation matrix from the camera coordinate system to the world coordinate system, ( x j , y j , z j ) is the world coordinate of the graticule.

4. The road measurement method based on image processing according to claim 1, wherein: The target marking coordinates are fused to obtain a road measurement curve, including: The target marking line coordinates are smoothed and then fitted to obtain the road measurement curve.

5. The road measurement method based on image processing according to claim 1, further comprising: determining a road bend based on a curve portion in the road measurement curve; A curve radius is determined based on the curvature of the curve portion to obtain a turning radius at a curve of the road.

6. A road measurement device based on image processing, comprising: An acquisition module, configured to acquire a road image including road markings based on a preset sampling period, wherein adjacent road images include an overlapping area of ​​a preset size; An extraction module, used for extracting the pixel coordinates of the road markings in the road image in the camera coordinate system; A first coordinate conversion module, configured to convert the pixel coordinates of the marking line into the world coordinates of the marking line in a world coordinate system based on the internal matrix parameters and the external matrix parameters of the camera coordinate system; The second coordinate conversion module is used to convert all the reticle world coordinates into target reticle coordinates in the same world coordinate system based on the overlapping area, including: Determining a preset number of feature points from the overlapping area; Determining an intermediate rotation matrix and an intermediate translation matrix between adjacent road images based on the marking line world coordinates of the feature points in adjacent road images; Taking the first frame of road image as the starting point, based on the intermediate rotation matrix and the intermediate translation matrix, all the marking world coordinates are converted to the same world coordinate system to obtain the corresponding target marking coordinates, including: the i+1 frame of road image is converted to the i-th frame of road image in the world coordinate system of the i-th intermediate marking coordinate is d i = R i · d i+1 + T i ,in, d i+1 is the world coordinate of the marking line of the i+1th frame road image, R i is the intermediate rotation matrix between the i+1th frame road image and the ith frame road image, T i is the intermediate translation matrix between the i+1th frame road image and the ith frame road image; i=0, 1, 2, ..., i is a natural number; sequentially calculate the i-1th intermediate marking line coordinates in the world coordinate system of the i-1th frame road image converted to the i+1th frame road image until i=0, and obtain the target marking line coordinates; The fusion module is used to fuse the target marking coordinates to obtain a road measurement curve.

7. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 5.

Images