Map updating method based on data fusion, storage medium and vehicle-mounted map device
Through the fusion processing of the on-board vision unit and satellite image data, the problem of inaccurate map updates in complex environments is solved, and more accurate and reliable map data updates are achieved, suitable for vehicle navigation and path planning.
Patent Information
- Application Number
- CN202410848439.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, the resolution and accuracy of satellite image data are not sufficient to capture road changes and building changes, especially in dense urban areas or complex terrain, resulting in inaccurate map information and bad weather may limit the update of map information.
Using a map update method based on data fusion, the image data during vehicle driving is obtained through the vehicle's on-board vision unit for fusing processing, and using techniques such as sharpening filtering, distortion correction, multi-scale wavelet transformation and weighted average calculation to generate accurate map data.
Improve the accuracy and reliability of map data, avoid the impact of bad weather and complex terrain on map updates, and ensure the accuracy of vehicle navigation and path planning.
Smart Images

Figure CN118960761A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent driving, and in particular relates to a map updating method based on data fusion, a storage medium and a vehicle-mounted map device. Background Art
[0002] Whether it is vehicle autonomous driving or assisted driving, the vehicle side needs to have accurate map information, and geographic data is constantly changing. For example, road maintenance, infrastructure, etc. will affect geographic data, which requires the vehicle-side map information to have accurate update capabilities; the commonly used source of updated map information is satellite image data, but the resolution and accuracy of satellite image data are sometimes not enough to capture all subtle changes in roads or buildings, especially in densely populated urban areas or complex terrain, which may cause inaccurate map information. In addition, satellite image data is affected by factors such as weather and clouds, and it may not be possible to obtain map information around the clock, especially in bad weather. Updates may be restricted. For this reason, a map update method based on data fusion is proposed. Summary of the invention
[0003] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a map updating method, a storage medium and a vehicle-mounted map device based on data fusion, which solve the problems in the prior art.
[0004] The purpose of the present invention can be achieved through the following technical solutions:
[0005] In a first aspect, a map updating method based on data fusion comprises the following steps:
[0006] Acquire a first image of the road condition during vehicle driving by using a vehicle-mounted vision unit, and pre-process the first image to obtain a second image;
[0007] Obtaining vehicle-mounted GPS positioning information, and obtaining a third image from a satellite image database according to the vehicle-mounted GPS positioning information;
[0008] The second image and the third image are fused to obtain a fourth image;
[0009] The map data on the vehicle side is updated using the fourth image.
[0010] Furthermore, the step of preprocessing the first image includes: processing the first image using a sharpening filter method, and the processing steps include:
[0011] Blur the first image using a Gaussian filter:
[0012]
[0013] Where G(x, y) is the output image pixel value of the Gaussian filter, that is, the pixel value of the blurred image; I(x, y) is the pixel value of the first image; σ is the standard deviation of the Gaussian filter; (x, y) is the pixel position in the image;
[0014] Calculating the difference between the pixels of the first image and the pixels of the blurred image to extract detail information in the first image;
[0015] Add the extracted detail information to the first image:
[0016] I sharp (x,y)=I(x,y)+α×D(x,y)
[0017] In the formula, I sharp (x, y) is the pixel value of the enhanced image; α is the enhancement coefficient, which is used to control the influence of detail information on the image; α is a value between 0 and 1, indicating the proportion of detail information.
[0018] Furthermore, the step of preprocessing the first image further includes correcting the distortion in the first image, and the correction step includes:
[0019] Selecting a radial distortion model to represent the distortion in the first image, and estimating distortion parameters in the first image by minimizing the residual of the radial distortion model;
[0020] Using the estimated distortion parameters, a correction map is calculated to map pixels in the distorted image to corresponding positions in the corrected image;
[0021] According to the correction map, the image is resampled to generate a corrected image; during the resampling process, a bilinear interpolation algorithm is used to estimate the corrected pixel values:
[0022]
[0023] In the formula, I corrected (u c , v c ) is the pixel (u c , v c ) is an estimated value; original (u original
[0024] +i,v original +j) is the pixel (u original +i,v original +j) value, where u original and v original is the pixel coordinate before correction, i and j are the offsets of the four nearest neighbors around the pixel before correction; wi and w j is the weight of bilinear interpolation, according to u c and v c with u original and v original Calculate the distance.
[0025] Furthermore, obtaining the vehicle-mounted GPS positioning information and obtaining the third image from the satellite image database according to the vehicle-mounted GPS positioning information includes: obtaining the longitude and latitude coordinates of the first image according to the vehicle-mounted GPS, and retrieving and obtaining the satellite image data of the corresponding position, i.e., the third image, from the satellite image database according to the longitude and latitude coordinates.
[0026] Furthermore, the step of fusing the second image with the third image includes:
[0027] Performing multi-scale transformation on the second image and the third image by using two-dimensional discrete wavelet transform;
[0028] In the multi-scale domain, the second image and the third image are decomposed into low-frequency components and high-frequency components;
[0029] fusing different components of the second image and the third image using a weighted average operation;
[0030] The fused image components are recombined to obtain a fourth image.
[0031] Furthermore, the expression for fusing different components of the second image and the third image using weighted average operation is:
[0032] F fused (x,y)=β·F1(x,y)+(1-β)·F2(x,y)
[0033] In the formula, F fused (x, y) is the pixel value of the fourth image after fusion at the coordinate (x, y); F1(x, y) and F2(x, y) are the pixel values of the second image and the third image to be fused at the same coordinates respectively; β is the weighting coefficient, and its value range is [0, 1].
[0034] Furthermore, the step of updating the map data on the vehicle side using the fourth image includes:
[0035] Use the nearest neighbor algorithm to match the fourth image with the features in the vehicle-side map.
[0036] Use the ORB algorithm to detect key points in the fourth image, calculate the descriptor of each key point, match the ORB descriptor in the fourth image with the ORB descriptor in the vehicle-side map, and find the corresponding map features; and filter the matching results to remove incorrect matching points;
[0037] The matched map feature information is updated into the vehicle-side map database to complete the update of the vehicle-side map.
[0038] Furthermore, in the nearest neighbor algorithm, the Euclidean distance is used to measure the F of the features in the fourth image. i Features of the car-side map F j The similarity between them is expressed as:
[0039]
[0040] Where n is the dimension of the feature vector, F i [m] and F j [m] are the eigenvectors F i and F j The mth component in .
[0041] In a second aspect, a computer storage medium stores a readable program, which can execute the above-mentioned map updating method based on data fusion when the program is running.
[0042] In a third aspect, a vehicle-mounted map device includes:
[0043] On-board visual module: including on-board cameras, driving recorders and other image acquisition devices, which can capture detailed information such as road conditions, traffic signs, and traffic facilities in real time during the vehicle's driving process;
[0044] Image processing and fusion module: including steps such as image registration, image transformation, and image enhancement to ensure that images from two different sources can be accurately aligned and fused;
[0045] Map update module: using the map update method based on data fusion to update the on-vehicle map data;
[0046] Data storage and management module: used to store and manage map data and related image data; user interface and interaction module: including display module and human-computer interaction module.
[0047] Beneficial effects of the present invention:
[0048] The vehicle-side map information is updated by combining the image data captured in real time by the on-board visual component with the satellite image data, avoiding the defect of inaccurate map information caused by the satellite image data being affected by complex buildings, terrain and bad weather. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0050] Figure 1 is a flow chart of a map updating method according to an embodiment of the invention;
[0051] Figure 2 It is a schematic diagram of the structure of the vehicle-mounted map device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] like Figure 1 As shown, a map updating method based on data fusion includes the following steps:
[0054] S1, obtaining a first image of a road condition during vehicle driving through a vehicle-mounted vision unit, and preprocessing the first image to obtain a second image;
[0055] The step of preprocessing the first image includes:
[0056] S11, using a sharpening filter method to perform clarity enhancement processing on the first image to improve the clarity of the image; specifically including:
[0057] 1) Use a Gaussian filter to blur the first image, which can remove some high-frequency details and make the image smoother;
[0058] The formula for blurring an image using a Gaussian filter can be expressed as:
[0059]
[0060] Where G(x, y) is the output image pixel value of the Gaussian filter, that is, the pixel value of the blurred image; I(x, y) is the pixel value of the first image; σ is the standard deviation of the Gaussian filter; (x, y) is the pixel position in the image. The expression is a two-dimensional Gaussian distribution function, which has a weight at each pixel position in the image. By moving the filter over the entire image and convolving the filter with the image, the blurred image pixel value G(x, y) can be obtained.
[0061] 2) Calculate the difference between the pixels of the first image and the pixels of the blurred image to extract the detail information in the first image; the formula for extracting the detail information is:
[0062] D(x,y)=I(x,y)-G(x,y)
[0063] Where D(x, y) is the detail information.
[0064] 3) The extracted detail information is added to the first image to enhance the sharpness of the image; this step usually amplifies the detail information to ensure that they have a significant impact on the final image; the calculation expression for the enhanced sharpness of the first image is:
[0065] I sharp (x,y)=I(x,y)+α×D(x,y)
[0066] In the formula, I sharp (x, y) is the pixel value of the enhanced image; α is the enhancement coefficient, which is used to control the influence of detail information on the image; α is a value between 0 and 1, indicating the proportion of detail information; the larger α is, the more obvious the influence of detail information on the image is, and the higher the degree of image sharpness enhancement is.
[0067] S12, correcting the distortion and distortion in the first image to ensure that the image accurately reflects the real scene;
[0068] The specific steps include:
[0069] 1) Selecting a radial distortion model to represent the distortion in the first image;
[0070] 2) estimating distortion parameters in the first image, including radial distortion coefficients k1, k2, k3;
[0071] The radial distortion parameters are estimated by minimizing the residual of the radial distortion model. The residual is calculated as follows:
[0072]
[0073] distortion(·)=(1+k1r 2 +k2r 4 +k3r 6 )
[0074] Where r is the residual of the radial distortion model, (u o , v o ) is the center point of the corrected image, (u d , v d) is the pixel coordinate in the distorted image, f is the focal length of the camera, and distortion(·) is the radial distortion model function.
[0075] 3) Using the estimated distortion parameters, calculate the correction map to map the pixels in the distorted image to the corresponding positions in the corrected image; the calculation formula of the correction map is:
[0076]
[0077] Among them, undistort(·) is the correction mapping function.
[0078] 4) According to the correction mapping, the image is resampled to generate a corrected image; in the resampling process, a bilinear interpolation algorithm is usually used to estimate the corrected pixel value, and the estimation formula is:
[0079]
[0080] In the formula, I corrected (u c , v c ) is the pixel (u c , v c ) is an estimated value; original (u original
[0081] +i,v original +j) is the pixel (u original +i,v original +j) value, where u original and v original is the pixel coordinate before correction, i and j are the offsets of the four nearest neighbors around the pixel before correction; w i and w j is the weight of bilinear interpolation, according to u c and v c with u original and v original Calculate the distance.
[0082] S2, obtaining the vehicle-mounted GPS positioning information, and obtaining the third image from the satellite image database according to the vehicle-mounted GPS positioning information; including: obtaining the longitude and latitude coordinates of the first image according to the vehicle-mounted GPS, and retrieving and obtaining the satellite image data of the corresponding position, i.e., the third image, from the satellite image database according to the longitude and latitude coordinates.
[0083] S3, fusing the second image and the third image to obtain a fourth image;
[0084] The step of fusing the first image data with the satellite image data comprises:
[0085] S31, using two-dimensional discrete wavelet transform to perform multi-scale transformation on the second image and the third image, so that the images can be represented and processed at different scales, thereby extracting richer information; the formula of two-dimensional discrete wavelet transform is:
[0086]
[0087] In the formula, X j,k is the wavelet coefficient with scale j and position k; s(n) is the discrete sampling of the original image (or signal); ψ j,k (n) is the wavelet basis function, its scale is determined by j and its position is determined by k; the wavelet basis function can be Daubechies wavelet, Haar wavelet, etc.
[0088] S32, decomposing the second image and the third image into low-frequency components and high-frequency components in a multi-scale domain;
[0089] For a two-dimensional image, wavelet transform is usually decomposed into sub-bands in the horizontal, vertical and diagonal directions, that is, low-frequency and high-frequency information in three directions. The decomposition formula is:
[0090]
[0091] In the formula, X j ,k,l represents the wavelet coefficients with scale j, horizontal position k and vertical position l; h(k) and g(k) are wavelet functions, which are used for low-pass and high-pass filtering in the horizontal and vertical directions respectively; LL j+1 , LH j+1 , HL j+1 and HH j+1 They represent the low-frequency, horizontal high-frequency, vertical high-frequency and diagonal high-frequency components of the next scale respectively. Through continuous iteration, the image can be decomposed into low-frequency and high-frequency components at different scales.
[0092] S33, using weighted average operation to fuse different components of the second image and the third image; the image fusion operation expression is:
[0093] F fused (x,y)=β·F1(x,y)+(1-β)·F2(x,y)
[0094] In the formula, F fused(x, y) is the pixel value of the fused image at the coordinate (x, y); F1(x, y) and F2(x, y) are the pixel values of the second image and the third image to be fused at the same coordinates; β is the weighting coefficient, which is used to control the weight distribution of the second image and the third image in the fusion, and the value range is [0, 1]; when β = 0, the fusion result is completely determined by the second image; when β = 1, it is completely determined by the first image. Each pixel of the second image and the third image is fused according to the principle of weighted average, and the weighting coefficient β determines the contribution of the second image and the third image to the fusion result.
[0095] S34, recombining the fused image components to obtain a fourth image.
[0096] The pixel values of the fused low-frequency components at the same coordinates and the pixel values of the fused high-frequency components at the same coordinates are added to obtain the pixel values of the final fused image at the coordinates (x, y), thereby obtaining the final fused image, i.e., the fourth image.
[0097] S4, using the fourth image to update the map data on the vehicle side. The updating steps include:
[0098] S41, using a nearest neighbor algorithm to match the fourth image with features in the vehicle-side map to ensure that they correspond in space;
[0099] The nearest neighbor algorithm is used for feature matching. Its basic idea is to compare a feature with all the features in another feature set and find the most similar feature; the Euclidean distance is used to measure: the F of the feature in the fourth image i Features of the car-side map F j The similarity between them is expressed as
[0100]
[0101] Where n is the dimension of the feature vector, F i [m] and F j [m] are the eigenvectors F i and F j The mth component in .
[0102] S42, extracting map features in the fourth image using an ORB algorithm detection algorithm;
[0103] Use the ORB algorithm to detect key points in the fourth image, calculate the descriptor of the descriptor of each key point, match the ORB descriptor in the fourth image with the ORB descriptor in the vehicle-side map, and find the corresponding map features; filter the matching results and remove erroneous matching points to ensure the accuracy and stability of the matching; the extracted map features include roads, buildings, traffic signs, etc.
[0104] It is worth mentioning that the ORB algorithm combines FAST key point detection and BRIEF descriptor generation technology, which has faster calculation speed and good performance.
[0105] S43, updating the extracted map feature information into the vehicle-side map database, completing the update of the vehicle-side map. The updated map data can be published to applications such as the vehicle navigation system for use in vehicle navigation and route planning.
[0106] The present invention also provides a computer-readable storage medium.
[0107] In one embodiment of the present invention, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the map updating method based on data fusion.
[0108] The invention also provides a vehicle-mounted map device.
[0109] In one embodiment of the present invention, Figure 2 The vehicle-mounted map device shown includes:
[0110] The vehicle-mounted vision module 100 includes a vehicle-mounted camera, a driving recorder and other image acquisition devices, which can capture detailed information such as road conditions, traffic signs, and traffic facilities during the driving process of the vehicle in real time;
[0111] Image processing and fusion module 200: including steps such as image registration, image transformation, and image enhancement to ensure that images from two different sources can be accurately aligned and fused;
[0112] Map updating module 300: using the map updating method based on data fusion to update the on-vehicle map data; that is, using the fused fourth image to update the internal map data on the vehicle;
[0113] Data storage and management module 400: used to store and manage map data and related image data;
[0114] User interface and interaction module 500: includes a display module and a human-computer interaction module. The display module is responsible for displaying map data, image data and other related information on the user interface. The human-computer interaction module provides interactive functions between the user and the system, such as map zooming, panning, searching, querying, etc.
[0115] The method of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.
[0116] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.
Claims
1. A map updating method based on data fusion, characterized in that: The following steps are involved: Acquire a first image of the road condition during vehicle driving by using a vehicle-mounted vision unit, and pre-process the first image to obtain a second image; Obtaining vehicle-mounted GPS positioning information, and obtaining a third image from a satellite image database according to the vehicle-mounted GPS positioning information; The second image and the third image are fused to obtain a fourth image; The map data on the vehicle side is updated using the fourth image.
2. A map updating method based on data fusion according to claim 1, characterized in that: The step of preprocessing the first image to obtain the second image includes: processing the first image using a sharpening filter method, and the processing steps include: 1) Blur the first image using a Gaussian filter: Where G(x, y) is the output image pixel value of the Gaussian filter, that is, the pixel value of the blurred image; I(x, y) is the pixel value of the first image; σ is the standard deviation of the Gaussian filter; (x, y) is the pixel position in the image; 2) Calculating the difference between the pixels of the first image and the pixels of the blurred image to extract detail information in the first image; 3) Add the extracted detail information to the first image: I sharp (x,y)=I(x,y)+α×D(x,y) In the formula, I sharp (x, y) is the pixel value of the enhanced image; α is the enhancement coefficient, which is used to control the influence of detail information on the image; α is a value between 0 and 1, indicating the proportion of detail information.
3. A map updating method based on data fusion according to claim 2, characterized in that: The step of preprocessing the first image to obtain the second image further includes correcting the distortion in the first image, and the correction step includes: Selecting a radial distortion model to represent the distortion in the first image, and estimating distortion parameters in the first image by minimizing the residual of the radial distortion model; Using the estimated distortion parameters, a correction map is calculated to map pixels in the distorted image to corresponding positions in the corrected image; According to the correction map, the image is resampled to generate a corrected image; during the resampling process, a bilinear interpolation algorithm is used to estimate the corrected pixel values: In the formula, I corrected (u c , v c ) is the pixel (u c , v c ) is an estimated value; original (u original +i,v original +j) is the pixel (u original +i,v original +j) value, where u original and v original is the pixel coordinate before correction, i and j are the offsets of the four nearest neighbors around the pixel before correction; w i and w j is the weight of bilinear interpolation, according to u c and v c with u original and v original Calculate the distance.
4. A map updating method based on data fusion according to claim 1 or 3, characterized in that The obtaining of the vehicle-mounted GPS positioning information and obtaining the third image from the satellite image database according to the vehicle-mounted GPS positioning information includes: obtaining the longitude and latitude coordinates of the first image according to the vehicle-mounted GPS, and retrieving and obtaining the satellite image data of the corresponding position from the satellite image database according to the longitude and latitude coordinates, that is, the third image.
5. A map updating method based on data fusion according to claim 4, characterized in that: The step of fusing the second image with the third image comprises: Performing multi-scale transformation on the second image and the third image by using two-dimensional discrete wavelet transform; In the multi-scale domain, the second image and the third image are decomposed into low-frequency components and high-frequency components; fusing different components of the second image and the third image using a weighted average operation; The fused image components are recombined to obtain a fourth image.
6. A map updating method based on data fusion according to claim 5, characterized in that: The expression for fusing different components of the second image and the third image using weighted average operation is: F fused (x,y)=β·F1(x,y)+(1-β)·F2(x,y) In the formula, F fused (x, y) is the pixel value of the fourth image after fusion at the coordinate (x, y); F1(x, y) and F2(x, y) are the pixel values of the second image and the third image to be fused at the same coordinates respectively; β is the weighting coefficient, and its value range is [0, 1].
7. A map updating method based on data fusion according to claim 6, characterized in that: The step of updating the map data on the vehicle side using the fourth image comprises: Use the nearest neighbor algorithm to match the fourth image with the features in the vehicle-side map. Use the ORB algorithm to detect key points in the fourth image, calculate the descriptor of each key point, match the ORB descriptor in the fourth image with the ORB descriptor in the vehicle-side map, and find the corresponding map features; and filter the matching results to remove incorrect matching points; The matched map feature information is updated to the vehicle-side map database to complete the update of the vehicle-side map.
8. A map updating method based on data fusion according to claim 7, characterized in that: In the nearest neighbor algorithm, the Euclidean distance is used to measure the F of the features in the fourth image. i Features of the car-side map F j The similarity between them is expressed as: Where n is the dimension of the feature vector, F i [m] and F j [m] are the eigenvectors F i and F j The mth component in .
9. A computer storage medium storing a readable program, characterized in that: When the program is running, it can execute the map updating method based on data fusion described in any one of claims 1 to 8.
10. A vehicle-mounted map device, characterized in that: include: On-board visual module: including on-board cameras, driving recorders and other image acquisition devices, which can capture detailed information such as road conditions, traffic signs, and traffic facilities in real time during the vehicle's driving process; Image processing and fusion module: including steps such as image registration, image transformation, and image enhancement to ensure that images from two different sources can be accurately aligned and fused; Map update module: using the map update method based on data fusion described in any one of claims 1 to 8 to update the vehicle map data; Data storage and management module: used to store and manage map data and related image data; User interface and interaction module: including display module and human-computer interaction module.
