Vehicle positioning method and device based on remote sensing of unmanned aerial vehicle, and electronic equipment
The remote sensing technology of the drone is obtained and the body information around the vehicle is combined with semantic segmentation and contour recognition, and the problem of insufficient vehicle positioning capabilities in autonomous driving without relying on high-precision maps is solved, and the vehicle is accurately and quickly positioned and road condition perception is achieved.
Patent Information
- Application Number
- CN202411079455.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-26
Smart Images

Figure CN119027836A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle positioning technology, and in particular to a vehicle positioning method, device and electronic equipment based on unmanned aerial vehicle remote sensing. Background Art
[0002] The existing commercial autonomous driving technologies are mainly divided into solutions that rely on high-precision maps and solutions that do not rely on high-precision maps. Among them, the solution based on high-precision maps relies on the rich road condition information and accurate road data information of high-precision maps, which can provide environmental perception assistance for the autonomous driving system, provide beyond-line-of-sight road condition information, and help the autonomous driving system complete path planning decisions; however, the current high-precision map solution has problems such as high construction and maintenance costs, slow map construction speed, etc., and it is difficult to popularize it to all-scenario roads.
[0003] At present, some autonomous driving technology suppliers and car companies have begun to pay attention to autonomous driving solutions that do not rely on high-precision maps. This solution has the characteristics of wide availability, strong generalization ability, and low maintenance cost. However, in solutions that do not rely on high-precision maps, there are more stringent requirements for vehicle perception and planning algorithms. However, in solutions that do not rely on high-precision maps, there are still problems such as insufficient vehicle positioning capabilities that make it impossible to accurately plan movement routes. Summary of the invention
[0004] In view of the shortcomings of the related technologies mentioned above, the present application provides a vehicle positioning method, device and electronic equipment based on drone remote sensing to solve the technical problem of insufficient vehicle positioning capability.
[0005] The present application provides a vehicle positioning method based on drone remote sensing, and the vehicle positioning method based on drone remote sensing includes:
[0006] In one embodiment of the present application, remote sensing image information is acquired based on a drone, the remote sensing image information includes a remote sensing image center and an original remote sensing image, and feature extraction is performed on the original remote sensing image to obtain remote sensing image features; vehicle body image information and vehicle body radar information are acquired, the vehicle body image information is a plurality of vehicle body surround images acquired by an on-board image collector of the vehicle, the vehicle body radar information is radar data around the vehicle body acquired by an on-board radar of the vehicle, and feature fusion is performed based on the vehicle body image information and the vehicle body radar information to generate overhead view image features, the overhead view image features are overhead view image features around the vehicle; semantic segmentation is performed on the remote sensing image features to obtain A remote sensing semantic segmentation image is obtained, contour recognition is performed on the remote sensing image features to obtain a remote sensing contour recognition image, and remote sensing geometric features of the vehicle are obtained based on the remote sensing semantic segmentation image and the remote sensing contour recognition image; semantic segmentation is performed on the overhead view image features to obtain an overhead semantic segmentation image, key pixel clusters are clustered on the overhead semantic segmentation image based on semantic classification and pixel distance, and the key pixel clusters are fitted to obtain the overhead geometric features of the vehicle; a relative coordinate position of the vehicle relative to the center of the remote sensing image is obtained based on a matching relationship between the remote sensing geometric features and the overhead geometric features, and the vehicle is positioned according to the relative coordinate position.
[0007] In one embodiment of the present application, generating overhead view image features based on feature fusion of the vehicle body image information and the vehicle body radar information includes: splicing the vehicle body image information and inputting the resultant into an image feature encoder to obtain vehicle body image features; mapping the vehicle body radar information into a preset three-dimensional space to obtain sparse point cloud data, and filtering the sparse point cloud data based on a preset density threshold to obtain a radar vector matrix; inputting the radar vector matrix into a radar feature encoder to obtain vehicle body radar features, the radar feature encoder including a fully connected network, a pooling network, and a convolutional neural network; the vehicle body image features and the vehicle body radar features are combined based on a feature fusion network to obtain the overhead view image features.
[0008] In one embodiment of the present application, combining the vehicle body image features and the vehicle body radar features based on a feature fusion network to obtain the overhead view image features includes: adding distance information to the vehicle body image features based on the vehicle body radar features to obtain overhead view prompt features with distance information; using the overhead view prompt features as key vectors and the vehicle body radar features as query vectors; inputting the query vector and the key vector into the feature fusion network for feature solution to obtain overhead view image features.
[0009] In one embodiment of the present application, extracting features from the original remote sensing image to obtain remote sensing image features includes: calculating the original remote sensing image based on a preset operator matrix to obtain image gradient information, wherein the image gradient information includes gradient amplitude and gradient direction; screening the image edge of the original remote sensing image using a non-maximum suppression method based on the image gradient information to obtain image edge information; aligning multiple original remote sensing images based on the center of the remote sensing image according to the image edge information to obtain an aligned original remote sensing image; and downsampling the aligned original remote sensing image to obtain the remote sensing image features.
[0010] In one embodiment of the present application, semantic segmentation is performed on the remote sensing image features to obtain a remote sensing semantic segmentation image, contour recognition is performed on the remote sensing image features to obtain a remote sensing contour recognition image, and the remote sensing geometric features of the vehicle are obtained based on the remote sensing semantic segmentation image and the remote sensing contour recognition image, including: semantic segmentation is performed on the remote sensing image features based on a road semantic segmentation network to obtain a remote sensing semantic segmentation image, the road semantic segmentation network includes four layers of semantic segmentation upsampling modules, each of the semantic segmentation upsampling modules includes a transposed convolution layer, an activation layer, and a pooling layer; contour recognition is performed on the remote sensing image features based on the road contour recognition network to obtain a remote sensing contour recognition image, the road contour recognition network includes four layers of contour recognition upsampling modules, each of the contour recognition upsampling modules includes a self-attention network, a transposed convolution layer, and a pooling layer. Based on the remote sensing semantic segmentation image and the remote sensing contour recognition image, remote sensing point features and remote sensing line features are obtained, and the remote sensing geometric features include the remote sensing point features and the remote sensing line features.
[0011] In one embodiment of the present application, semantic segmentation is performed on the top-view image features to obtain a top-view semantic segmentation image, the top-view semantic segmentation image is clustered based on semantic classification and pixel distance to obtain key pixel clusters, and the key pixel clusters are fitted to obtain the top-view geometric features of the vehicle, including: fitting the key pixel clusters to obtain a fitting contour of the top-view image features, the fitting contour including at least a line contour, a lamp pole contour, a lane line contour, a wall edge contour, and a road sign edge contour; based on the fitting contour and the top-view semantic segmentation image, top-view point features and top-view line features are obtained, and the top-view geometric features include the top-view point features and the top-view line features.
[0012] In one embodiment of the present application, the remote sensing geometric features include the remote sensing point features and the remote sensing line features, and the overhead geometric features include the overhead point features and the overhead line features; obtaining the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead geometric features includes: inputting the remote sensing point features and the overhead point features into a preset point feature matching formula to obtain point feature measurement values, and inputting the remote sensing line features and the overhead line features into a preset line feature matching formula to obtain line feature measurement values; obtaining an expression for the relative coordinate position based on the point feature measurement values and the line feature measurement values; and solving the expression for the relative coordinate position based on maximum likelihood estimation to obtain the relative coordinate position.
[0013] In one embodiment of the present application, obtaining the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the bird's-eye view geometric features further includes:
[0014]
[0015] in, The vehicle pose is x t The maximum likelihood estimate of x t Represents the relative coordinate position of the vehicle relative to the center of the remote sensing image, represents the point feature measurement value, represents the line feature measurement value.
[0016] The embodiment of the present application also provides a vehicle positioning device based on drone remote sensing, characterized in that the vehicle positioning device based on drone remote sensing includes: an image input module, used to obtain remote sensing image information based on a drone, the remote sensing image information includes the remote sensing image center and the original remote sensing image, and feature extraction is performed based on the original remote sensing image to obtain remote sensing image features; an image acquisition module, used to obtain vehicle body image information and vehicle body radar information, and obtain overhead image features based on the vehicle body image information and the vehicle body radar information; a remote sensing processing module, used to perform semantic segmentation on the remote sensing image features to obtain a remote sensing semantic segmentation image, and extract the remote sensing image features. The invention relates to a method for performing contour recognition on the features to obtain a remote sensing contour recognition image, and obtaining remote sensing geometric features based on the remote sensing semantic segmentation image and the remote sensing contour recognition image; a bird's-eye view processing module, used for performing semantic segmentation on the bird's-eye view image features to obtain a bird's-eye view semantic segmentation image, clustering the bird's-eye view semantic segmentation image based on semantic classification and pixel distance to obtain key pixel clusters, and fitting the key pixel clusters to obtain bird's-eye view geometric features; a vehicle positioning module, used for obtaining a relative coordinate position of a vehicle relative to the center of the remote sensing image based on a matching relationship between the remote sensing geometric features and the bird's-eye view geometric features, and positioning the vehicle according to the relative coordinate position.
[0017] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the vehicle positioning method based on drone remote sensing as described in any of the above embodiments.
[0018] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer executes a vehicle positioning method based on drone remote sensing as described in any of the above embodiments.
[0019] Beneficial effects of the present application: The embodiments of the present application provide a vehicle positioning method, device and electronic equipment based on drone remote sensing. The method obtains remote sensing image information based on a drone, extracts features from the original remote sensing image to obtain remote sensing image features, obtains vehicle body image information and vehicle body radar information, obtains overhead image features based on the vehicle body image information and vehicle body radar information, obtains remote sensing geometric features based on the remote sensing image features, semantically segments the overhead image features to obtain an overhead semantic segmentation image, clusters the overhead semantic segmentation image based on semantic classification and pixel distance to obtain key pixel clusters, fits the key pixel clusters to obtain overhead geometric features, and obtains the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead geometric features. The method provides that the remote sensing image is extracted from the drone end and then sent to the vehicle end, thereby reducing the bandwidth occupancy of vehicle-machine data transmission and improving the real-time performance of vehicle positioning. The relative coordinate position is obtained by geometric features to locate the vehicle, so as to achieve accurate and rapid positioning of the vehicle. The vehicle body radar information and vehicle body image information feature fusion are used to achieve accurate perception of road condition information around the vehicle body.
[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 is a schematic diagram of an implementation environment of a vehicle positioning method based on UAV remote sensing shown in an exemplary embodiment of the present application;
[0022] Figure 2 is a flow chart of a vehicle positioning method based on UAV remote sensing shown in an exemplary embodiment of the present application;
[0023] Figure 3 is a flowchart of remote sensing image feature extraction shown in an exemplary embodiment of the present application;
[0024] Figure 4 is a schematic diagram of a feature fusion process shown in an exemplary embodiment of the present application;
[0025] Figure 5 is a schematic diagram of a semantic segmentation network and a contour recognition network shown in an exemplary embodiment of the present application;
[0026] Figure 6 is a block diagram of a vehicle positioning device based on drone remote sensing shown in an exemplary embodiment of the present application;
[0027] Figure 7 It is a structural schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0028] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0029] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.
[0030] It should be noted that in this application, "first", "second", etc. are only used to distinguish similar objects, and are not used to limit the order or precedence of similar objects. The variations of "including", "having", etc. described above indicate that the scope covered by the subject of the word is not exclusive except for the examples shown by the word.
[0031] It is understood that the various numbers, step numbers, and other reference numerals recorded in this application are distinguished for the convenience of description and are not intended to limit the scope of this application. The size of the reference numerals in this application does not mean the order of execution. The execution order of each process should be determined by its function and internal logic.
[0032] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.
[0033] The embodiments of the present application respectively propose a vehicle positioning method based on drone remote sensing, a vehicle positioning device based on drone remote sensing, an electronic device, a computer-readable storage medium and a computer program product, which will be described in detail below.
[0034] See also Figure 1 , Figure 1 It is a schematic diagram of an implementation environment of a vehicle positioning method based on UAV remote sensing shown as an exemplary embodiment of the present application.
[0035] like Figure 1As shown, the implementation environment may include a drone end 101 and a vehicle end 102, wherein the drone end 101 acquires remote sensing image information, the remote sensing image information includes a remote sensing image center and multiple original remote sensing images, and performs feature extraction based on the original remote sensing images to obtain remote sensing image features. The drone end sends the extracted remote sensing image features to the vehicle end 102, so that the vehicle end 102 performs semantic segmentation and contour recognition on the remote sensing image features to obtain remote sensing geometric features. The vehicle end 102 is also used to acquire vehicle body image information and vehicle body radar information, and obtain overhead view image features based on the vehicle body image information and vehicle body radar information, and obtain overhead view geometric features based on the overhead view image features, and then obtain the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead view geometric features, and locate the vehicle based on the relative coordinate position.
[0036] See also Figure 2 , Figure 2 is a flowchart of a vehicle positioning method based on drone remote sensing, which is shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is as follows. The method may also be applicable to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.
[0037] like Figure 2 As shown, in an exemplary embodiment, the vehicle positioning method based on drone remote sensing includes at least steps S210 to S250, which are described in detail as follows:
[0038] Step S210, acquiring remote sensing image information based on the UAV, the remote sensing image information including the remote sensing image center and the original remote sensing image, performing feature extraction based on the original remote sensing image to obtain remote sensing image features.
[0039] In one embodiment of the present application, the drone includes a visible light sensor and several single spectrum sensors provided by the multispectral imaging system, and the physical location Loc of each sensor in the drone coordinate system is obtained from the drone operation manual. i ={x i ,y i}, where i represents sensors at different positions. The center point of the image collected by the near-infrared spectral camera in the drone is set as the center of the remote sensing image, that is, the center coordinates of the original remote sensing image collected by the near-infrared camera are set as the center of the remote sensing image ReCo NIR ={0,0}, then the pixel deviation value ReCo of the original remote sensing image collected by the other cameras can be calculated based on the center of the remote sensing image i =Loc i -ReCo NIR , thereby realizing the phase difference correction between different sensors of the drone.
[0040] In one embodiment of the present application, extracting features from an original remote sensing image to obtain remote sensing image features includes: calculating the original remote sensing image based on a preset operator matrix to obtain image gradient information, the image gradient information including gradient amplitude and gradient direction; screening the image edge of the original remote sensing image using a non-maximum suppression method based on the image gradient information to obtain image edge information; aligning multiple original remote sensing images based on the center of the remote sensing image according to the image edge information to obtain an aligned original remote sensing image; and downsampling the aligned original remote sensing image to obtain remote sensing image features.
[0041] In one embodiment of the present application, the collected original remote sensing image is first smoothed by using a filtering method (such as Gaussian filtering or histogram smoothing). Then, the image gradient information of the smoothed original remote sensing image is calculated based on a preset Sobel operator, and the calculation is as follows:
[0042] G x =sobel x *A Formula (1)
[0043] G y =sobel y *A Formula (2)
[0044]
[0045] In formula (1), formula (2), formula (3), and formula (4), G x G is the calculation result of the preset first operator matrix on the image. y is the calculation result of the preset second operator matrix on the image, A is the pixel matrix of the original remote sensing image, G is the gradient amplitude value, θ is the gradient direction, where
[0046] In one embodiment of the present application, after the gradient amplitude value and gradient direction of the original remote sensing image are calculated, the non-maximum suppression (NMS) method is used to screen the real edge of the image. The screening method includes: comparing the gradient values of the central pixel of the original remote sensing image and the surrounding pixel points point by point in the four directions of right, top, upper right, and upper left on the pixel matrix of the original remote sensing image. If the gradient value of the point is the largest, it is retained, otherwise it is set to 0. Finally, a double threshold is used to process continuous edges, and the noise points remaining by non-maximum suppression are removed to obtain image edge information. Multiple original remote sensing images obtained by multiple drone sensors are screened in this way to obtain the image edge information of each original remote sensing image.
[0047] In one embodiment of the present application, multiple original remote sensing images are aligned based on the center of the remote sensing image according to the image edge information, and the aligned original remote sensing image includes: using the original remote sensing image captured by the near infrared camera as the template image I NIR (x), the center of the original remote sensing image captured by the near-infrared camera is taken as the center of the remote sensing image, and the images acquired by the other cameras are the distorted images I i (y), where x = [x1, x2] T is the image center coordinate of the template image, that is, the center of the remote sensing image, is the image center coordinate of the distorted image, and the edge point of the template image is used as the target pixel point τ={x k ,k=1,2,…,K}, the corresponding relationship of finding the target pixel in other distorted images can be expressed by y=φ(x;p), where p=[p1,p2,…,p N ] t To map the parameter vector of the transformation, the optimization result of the parameter vector p is solved by solving the minimization problem of the difference between the template image and the distorted image. Assuming that the brightness received by each camera is constant, the above image alignment problem can be transformed into a parameter estimation problem:
[0048] minE(p)=min∑ x∈τ |I r (x)-Ψ(I ω (φ(x;p)),α)| 2 Formula (5)
[0049] Among them I r represents the template image, I ω represents the distorted image, and Ψ(I,α) is the preset brightness conversion formula, which is used to reduce the irrelevance caused by brightness transformation between images. After solving the parameter vector p of the optimal mapping transformation according to the above, the affine matrix can be established to achieve subsequent image alignment.
[0050] In one embodiment of the present application, the aligned original remote sensing image is downsampled to obtain remote sensing image features such as Figure 3 As shown, Figure 3This is a flowchart of remote sensing image feature extraction shown in an exemplary embodiment of the present application. The downsampling feature extraction method uses a four-layer progressive convolutional network module to extract remote sensing image features. Each convolutional network module consists of a convolution layer, a pooling layer, an activation layer, and a channel-spatial attention fusion network layer (CBAM). The convolution layer uses a convolution kernel of size 3×3, and the number of convolution kernels is 64, 128, 256, and 512 with the network depth. In each layer, the image after convolution, pooling, and activation processing uses CBAM to focus on important information in the image features. In this embodiment, the important information includes but is not limited to vehicle information, road information, and traffic signal information. After extracting the remote sensing image features, the remote sensing image features are transmitted to the vehicle end using the data transmission module for vehicle positioning.
[0051] Step S220, obtaining vehicle body image information and vehicle body radar information, wherein the vehicle body image information is a plurality of vehicle body surround images collected by the vehicle-mounted image, and the vehicle body radar information is radar data around the vehicle body collected by the vehicle-mounted radar, and feature fusion is performed based on the vehicle body image information and the vehicle body radar information to obtain the top view image features.
[0052] In one embodiment of the present application, vehicle body image information is acquired through a plurality of surround-view cameras surrounding the vehicle body, vehicle body radar information is acquired through a radar sensor deployed on the vehicle body, and top-view image features are acquired based on the vehicle body image information and the vehicle body radar information.
[0053] In one embodiment of the present application, obtaining overhead view image features based on vehicle body image information and the vehicle body radar information includes: splicing multiple vehicle body image information and inputting them into an image feature encoder to obtain vehicle body image features; mapping the vehicle body radar information to a preset three-dimensional space to obtain sparse point cloud data, screening the sparse point cloud data based on a preset density threshold to obtain a radar vector matrix containing depth information. In this embodiment, the portion of the sparse point cloud data that exceeds the density threshold is regarded as data containing depth information, and a radar vector matrix containing depth information is generated based on the point cloud data that exceeds the density threshold in the sparse point cloud. The radar vector matrix is input into a radar feature encoder to obtain vehicle body radar features, and the radar feature encoder includes a fully connected network, a pooling network, and a convolutional neural network; the vehicle body image features and the vehicle body radar features are combined based on a feature fusion network to obtain overhead view image features.
[0054] In one embodiment of the present application, multiple vehicle body images captured by multiple surround view cameras are stitched together to obtain a stitched image input dimension of: 3N×H×W, where N represents the number of surround view cameras, H represents the image height captured by the camera, and W represents the image width captured by the camera. The stitched image is input into an image encoder, which can be a convolution-based ResNet network, an attention-based Vision Transformer (ViT) network, or a self-supervised learning-based DINO network. After the stitched image is processed by the graphic encoder, a vehicle body image feature f having F channels is output. vis , whose size is F×H×W.
[0055] In one embodiment of the present application, vehicle body radar information is obtained by a millimeter wave radar, and the vehicle body radar information is mapped to a preset three-dimensional space to obtain sparse point cloud data. Therefore, a density threshold M needs to be preset to screen volume units whose point cloud density meets specific requirements in the preset three-dimensional space. That is, if the number of radar point clouds contained in a certain volume unit exceeds the preset density threshold M, it is regarded as a volume unit containing key depth information, and the volume units containing the key depth information are extracted. The number of extractions is set to N, thereby obtaining an N×D radar vector matrix, where D is the data attribute vector length of each radar point (generated by a one-hot encoding method), and N represents the number of radar samples in the screened volume unit.
[0056] In one embodiment of the present application, a radar vector matrix is input into a radar feature encoder to encode the radar vector matrix to mine deep information in radar data to obtain a vehicle body radar feature. The encoding operation of the radar vector matrix includes: first, the radar vector matrix is encoded voxel by voxel in discrete three-dimensional space through a fully connected network (FCN) (the input feature size is X×Y×Z), and then a 1×1 pooling network is used to process the encoded features to integrate and form a high-dimensional feature matrix of size F×X×Y×Z, where F represents the feature vector length of a single voxel point, and finally a vehicle body radar feature f of size F×X×Y is generated through a 1×1 convolutional neural network. rad .
[0057] In one embodiment of the present application, combining the vehicle body image features and the vehicle body radar features based on a feature fusion network to obtain a top view image feature includes: adding distance information to the vehicle body image features based on the vehicle body radar features to obtain a top view prompt feature with distance information; using the top view prompt feature as a key vector and the vehicle body radar feature as a query vector; inputting the query vector and the key vector into the feature fusion network for feature solution to obtain the top view image feature.
[0058] In one embodiment of the present application, the process of fusing the vehicle body radar features and the vehicle body image features is as follows: Figure 4 As shown, Figure 4 is a schematic diagram of a feature fusion process shown in an exemplary embodiment of the present application. First, a convolutional neural network is used to transform the F×H×W vehicle body image feature f vis Transformed into an image feature vector f′ of the form X×Y×Z vis , in order to achieve the complete distribution of the vehicle body image features in three-dimensional space. Then a convolution layer with F 1×1 convolution kernels is used to transform the image feature vector f′ vis Transformed into an image feature vector f″ of the form F×X×Y vis , and then the image feature vector f″ vis With the vehicle body radar signature f rad Input the cross attention network respectively to generate the top view prompt feature Q containing radar distance information L .
[0059] In one embodiment of the present application, the image feature vector f″ vis With the vehicle body radar signature f rad Input the cross attention network respectively to generate the top view prompt feature Q containing radar distance information L Including: image feature vector f″ vis As input, they are respectively compared with the preset linear transformation matrix W q ( Figure 4 Medium green square) and W v ( Figure 4 The purple square in the middle performs a dot product to generate the query vector q and value vector v in the cross attention calculation, with the vehicle body radar feature f rad With the preset linear change matrix W k ( Figure 4 The dot product of the blue square in the figure is used as the keyword vector k in the cross attention calculation, and then a query point position matrix q is initialized in the form of the query vector q pos , vector q, k, v and query point position matrix q pos At the same time, input the cross attention network to obtain the top view prompt feature Q L :
[0060]
[0061] In formula (6), N represents the number of preset sampling points, q pos represents the query point position matrix, q represents the query vector, v represents the value vector, k represents the keyword vector, Q L Indicates the top view hint feature.
[0062] In one embodiment of the present application, the top view prompt feature Q with distance information is L As the key value vector, the vehicle body radar feature f rad As the query vector, it is input into the feature fusion network for feature solution to obtain the top view image feature f bev , the calculation principle is as follows:
[0063]
[0064] In formula (7), f bev is the top view image feature, f rad is the vehicle body radar feature, Q L is the top view hint feature. Through the above self-attention calculation, the top view feature vector f containing depth information is obtained. bev , whose eigenvector size is F×X×Y.
[0065] Step S230, semantically segmenting the remote sensing image features to obtain a remote sensing image semantic segmentation image, performing contour recognition on the remote sensing image features to obtain a remote sensing contour recognition image, and obtaining remote sensing geometric features of the vehicle based on the remote sensing semantic segmentation image and the remote sensing contour recognition image.
[0066] In one embodiment of the present application, semantic segmentation is performed on remote sensing image features to obtain a remote sensing semantic segmentation image, contour recognition is performed on remote sensing image features to obtain a remote sensing contour recognition image, and remote sensing geometric features are obtained based on the remote sensing semantic segmentation image and the remote sensing contour recognition image, including: semantic segmentation is performed on remote sensing image features based on a road semantic segmentation network to obtain a remote sensing semantic segmentation image, the road semantic segmentation network includes four layers of semantic segmentation upsampling modules, each semantic segmentation upsampling module includes a transposed convolution layer, an activation layer, and a pooling layer; contour recognition is performed on remote sensing image features based on a road contour recognition network to obtain a remote sensing contour recognition image, the road contour recognition network includes four layers of contour recognition upsampling modules, each contour recognition upsampling module includes a self-attention network, a transposed convolution layer, and a pooling layer. Based on the remote sensing semantic segmentation image and the remote sensing contour recognition image, remote sensing point features and remote sensing line features are obtained, and remote sensing geometric features include remote sensing point features and remote sensing line features.
[0067] In one embodiment of the present application, semantic segmentation and contour recognition are performed on remote sensing image features. Figure 5 As shown, Figure 5This is a schematic diagram of a semantic segmentation network and a contour recognition network shown in an exemplary embodiment of the present application. The remote sensing image features are input into the semantic segmentation network. The semantic segmentation upsampling module doubles the size of the input feature map and reduces the channel by two times, and passes it to the next layer of semantic segmentation upsampling module. For the output semantic segmentation upsampling module, a fully connected network and an activation layer are used for final processing, and the remote sensing image features are divided into N categories pixel by pixel, each of which represents a semantic meaning. In this embodiment, the remote sensing image features are divided into roads, traffic facilities, lawns, houses, woods, and waters. Similarly, after the remote sensing image features are input into the contour recognition network, the feature map size is gradually expanded and the number of feature map channels is reduced through operations such as convolution layers. Finally, the output result of the contour recognition upsampling module is mapped to two channels using a 1×1 convolution layer and an activation layer, and a binary image is generated to depict the remote sensing contour recognition image. After obtaining the remote sensing semantic segmentation image and the remote sensing contour recognition image, the remote sensing point features and remote sensing line features of the remote sensing image features are obtained based on the remote sensing semantic segmentation image and the remote sensing contour recognition image.
[0068] Step S240, semantically segmenting the top view image features to obtain a top view semantic segmentation image, clustering the top view semantic segmentation image based on semantic classification and pixel distance to obtain key pixel clusters, and fitting the key pixel clusters to obtain top view geometric features of the vehicle.
[0069] In one embodiment of the present application, semantic segmentation is performed on overhead image features to obtain a overhead semantic segmentation image, the overhead semantic segmentation image is clustered based on semantic classification and pixel distance to obtain key pixel clusters, and the key pixel clusters are fitted to obtain overhead geometric features, including: fitting the key pixel clusters to obtain a fitting contour of the overhead image features, the fitting contour including at least a line contour, a lamp pole contour, a lane line contour, a wall edge contour, and a road sign edge contour; based on the fitting contour and the overhead semantic segmentation image, overhead point features and overhead line features are obtained, and the overhead geometric features include overhead point features and overhead line features.
[0070] Step S250, obtaining the relative position coordinates of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the bird's-eye view geometric features, and positioning the vehicle according to the relative position coordinates.
[0071] In one embodiment of the present application, obtaining the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead geometric features includes: inputting the remote sensing point features and the overhead point features into a preset point feature matching formula to obtain point feature measurement values, and inputting the remote sensing line features and the overhead line features into a preset line feature matching formula to obtain line feature measurement values, the remote sensing geometric features include remote sensing point features and remote sensing line features, and the overhead geometric features include overhead point features and overhead line features; obtaining an expression for the relative coordinate position based on the point feature measurement values and the line feature measurement values; and solving the expression for the relative coordinate position based on maximum likelihood estimation to obtain the relative coordinate position.
[0072] In one embodiment of the present application, a three-degree-of-freedom is set to describe the two-dimensional posture attribute x of the vehicle. t =[x t ,y t ,θ t ], where x t ,y t represents the relative coordinate position of the vehicle relative to the center point of the remote sensing image, θ t represents the vehicle's driving angle; the geometric features of the vehicle's surrounding environment at time t are set as Z t ={z p ,z l}, where z p is the feature of the top view point, z l is the feature of the bird’s-eye view. The geometric features of road condition information in remote sensing images are defined by M = {m p ,m l} means, where m p is the remote sensing point feature, m l is the remote sensing line feature. Then the relationship between the overlooking point feature and the remote sensing point feature can be defined as c = [c p ,c l ], where c p ={i,j} represents the correlation between the i-th point feature in the overhead image feature and the j-th point feature in the remote sensing image feature. Similarly, c l ={m,n} represents the correlation between the line features of the overhead image features and the remote sensing image features. According to the above definition, the remote sensing line features and the overhead line features are input into the preset line feature matching formula to obtain the line feature measurement value, including:
[0073]
[0074] In formula (8), (m,n)∈c l Represents the correspondence of line features, Indicates the vehicle position as x t The nth line feature measurement value in the case of It is the preset linear influence parameter.
[0075] Similarly, inputting the remote sensing point features and the overlooking point features into the preset point feature matching formula to obtain the point feature measurement values includes:
[0076]
[0077] In formula (9), (i,j)∈c p Represents the correspondence between point features, Indicates the vehicle position as x t The nth point feature measurement value in the case of To preset point influence parameters.
[0078] According to equations (8) and (9), the expression of relative coordinate position is obtained:
[0079]
[0080] In formula (10), The vehicle pose is x t The maximum likelihood estimate of Z t is the geometric feature measurement value, Z t Based on the point feature measurement value and line feature measurement value, x t Represents the relative coordinate position of the vehicle relative to the center of the remote sensing image, represents the point feature measurement value, Represents line feature measurements.
[0081] In one embodiment of the present application, deep learning, ant colony or simulated annealing algorithms can be used to calculate x in formula (10): t The optimal solution is to t The optimal solution is used as the relative coordinate position of the vehicle relative to the center of the remote sensing image at time t, and then the GNSS and combined inertial navigation IMU positioning system are used to calculate the final positioning coordinates of the vehicle in the world coordinate system based on the relative coordinate position, so as to realize the positioning of the vehicle based on UAV remote sensing.
[0082] See also Figure 6 , Figure 6 is a block diagram of a vehicle positioning device based on drone remote sensing, shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown is as follows. The device may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.
[0083] like Figure 6As shown, the exemplary vehicle positioning device based on UAV remote sensing includes: an image input module 601, an image acquisition module 602, a remote sensing processing module 603, a bird's-eye view processing module 604 and a vehicle positioning module 605.
[0084] The image input module 601 is used to obtain remote sensing image information based on the UAV, the remote sensing image information includes the remote sensing image center and the original remote sensing image, and extract features from the original remote sensing image to obtain remote sensing image features;
[0085] An image acquisition module 602 is used to obtain vehicle body image information and vehicle body radar information, and obtain overhead image features based on the vehicle body image information and vehicle body radar information;
[0086] The remote sensing processing module 603 is used to perform semantic segmentation on the remote sensing image features to obtain a remote sensing semantic segmentation image, perform contour recognition on the remote sensing image features to obtain a remote sensing contour recognition image, and obtain remote sensing geometric features based on the remote sensing semantic segmentation image and the remote sensing contour recognition image;
[0087] A bird's-eye view processing module 604 is used to perform semantic segmentation on bird's-eye view image features to obtain a bird's-eye view semantic segmentation image, cluster the bird's-eye view semantic segmentation image based on semantic classification and pixel distance to obtain key pixel clusters, and fit the key pixel clusters to obtain bird's-eye view geometric features;
[0088] The vehicle positioning module 605 is used to obtain the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the bird's-eye view geometric features, and to position the vehicle according to the relative coordinate position.
[0089] Figure 7 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0090] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method described in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0091] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.
[0092] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.
[0093] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0094] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0095] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0096] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the vehicle positioning method based on drone remote sensing as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.
[0097] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the vehicle positioning method based on drone remote sensing provided in each of the above embodiments.
[0098] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.
Claims
1. A vehicle positioning method based on UAV remote sensing, characterized in that: The vehicle positioning method based on UAV remote sensing includes: Acquiring remote sensing image information based on a drone, wherein the remote sensing image information includes a remote sensing image center and an original remote sensing image, and extracting features from the original remote sensing image to obtain remote sensing image features; Acquire vehicle body image information and vehicle body radar information of the vehicle, wherein the vehicle body image information is a plurality of vehicle body surround images acquired by an on-board image collector of the vehicle, and the vehicle body radar information is radar data around the vehicle body acquired by an on-board radar of the vehicle, and generate overhead view image features by performing feature fusion based on the vehicle body image information and the vehicle body radar information, wherein the overhead view image features are overhead view image features around the vehicle; Performing semantic segmentation on the remote sensing image features to obtain a remote sensing semantic segmentation image, performing contour recognition on the remote sensing image features to obtain a remote sensing contour recognition image, and obtaining remote sensing geometric features of the vehicle based on the remote sensing semantic segmentation image and the remote sensing contour recognition image; Performing semantic segmentation on the top-view image features to obtain a top-view semantic segmentation image, clustering the top-view semantic segmentation image based on semantic classification and pixel distance to obtain key pixel clusters, and fitting the key pixel clusters to obtain top-view geometric features of the vehicle; The relative coordinate position of the vehicle relative to the center of the remote sensing image is obtained based on the matching relationship between the remote sensing geometric features and the bird's-eye view geometric features, and the vehicle is positioned according to the relative coordinate position.
2. The vehicle positioning method based on UAV remote sensing according to claim 1 is characterized in that: The feature fusion based on the vehicle body image information and the vehicle body radar information to generate the top view image feature includes: The vehicle body image information is spliced and input into an image feature encoder to obtain a vehicle body image feature; Mapping the vehicle body radar information into a preset three-dimensional space to obtain sparse point cloud data, and filtering the sparse point cloud data based on a preset density threshold to obtain a radar vector matrix; Inputting the radar vector matrix into a radar feature encoder to obtain a vehicle body radar feature, wherein the radar feature encoder includes a fully connected network, a pooling network, and a convolutional neural network; The vehicle body image features and the vehicle body radar features are combined based on a feature fusion network to obtain the top view image features.
3. The vehicle positioning method based on UAV remote sensing according to claim 2 is characterized in that: Combining the vehicle body image feature and the vehicle body radar feature based on a feature fusion network to obtain the top view image feature includes: Adding distance information to the vehicle body image feature based on the vehicle body radar feature to obtain a top view prompt feature with distance information; Using the top view prompt feature as a key vector and the vehicle body radar feature as a query vector; The query vector and the key-value vector are input into the feature fusion network for feature solving to obtain the bird's-eye view image features.
4. The vehicle positioning method based on UAV remote sensing according to claim 1 is characterized in that: The remote sensing image features obtained by extracting features from the original remote sensing image include: Calculating the original remote sensing image based on a preset operator matrix to obtain image gradient information, wherein the image gradient information includes gradient amplitude and gradient direction; Based on the image gradient information, the image edge of the original remote sensing image is screened by adopting a non-maximum suppression method to obtain image edge information; Aligning multiple original remote sensing images based on the center of the remote sensing image according to the image edge information to obtain an aligned original remote sensing image; The aligned original remote sensing image is downsampled to obtain the remote sensing image features.
5. The vehicle positioning method based on UAV remote sensing according to claim 1 is characterized in that: Performing semantic segmentation on the remote sensing image features to obtain a remote sensing semantic segmentation image, performing contour recognition on the remote sensing image features to obtain a remote sensing contour recognition image, and obtaining the remote sensing geometric features of the vehicle based on the remote sensing semantic segmentation image and the remote sensing contour recognition image includes: Based on a road semantic segmentation network, the remote sensing image features are semantically segmented to obtain a remote sensing semantic segmentation image, wherein the road semantic segmentation network includes four layers of semantic segmentation upsampling modules, each of which includes a transposed convolution layer, an activation layer, and a pooling layer; Based on the road contour recognition network, the remote sensing image features are contour recognized to obtain a remote sensing contour recognition image, wherein the road contour recognition network includes four layers of contour recognition upsampling modules, each of which includes a self-attention network, a transposed convolution layer, and a pooling layer; Remote sensing point features and remote sensing line features are obtained based on the remote sensing semantic segmentation image and the remote sensing contour recognition image, and the remote sensing geometric features include the remote sensing point features and the remote sensing line features.
6. The vehicle positioning method based on UAV remote sensing according to claim 1 is characterized in that: The overhead image features are semantically segmented to obtain an overhead semantic segmentation image, the overhead semantic segmentation image is clustered based on semantic classification and pixel distance to obtain key pixel clusters, and the overhead geometric features of the vehicle are obtained by fitting the key pixel clusters. Fitting the key pixel clusters to obtain a fitting contour of the overhead image feature, wherein the fitting contour includes at least a line contour, a lamp pole contour, a lane line contour, a wall edge contour, and a road sign edge contour; Based on the fitting contour and the overhead semantic segmentation image, overhead point features and overhead line features are obtained, and the overhead geometric features include the overhead point features and the overhead line features.
7. The vehicle positioning method based on UAV remote sensing according to claim 1 is characterized in that: The remote sensing geometric features include the remote sensing point features and the remote sensing line features, and the overhead view geometric features include the overhead view point features and the overhead view line features; Obtaining the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead geometric features includes: Input the remote sensing point feature and the overlooking point feature into a preset point feature matching formula to obtain a point feature measurement value, and input the remote sensing line feature and the overlooking line feature into a preset line feature matching formula to obtain a line feature measurement value; Obtaining an expression for the relative coordinate position based on the point feature measurement value and the line feature measurement value; The expression of the relative coordinate position is solved based on the maximum likelihood estimation to obtain the relative coordinate position.
8. The vehicle positioning method based on UAV remote sensing according to claim 7 is characterized in that: Obtaining the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead geometric features also includes: in, The vehicle pose is x t The maximum likelihood estimate of x t Represents the relative coordinate position of the vehicle relative to the center of the remote sensing image, represents the point feature measurement value, represents the line feature measurement value.
9. A vehicle positioning device based on drone remote sensing, characterized in that: The vehicle positioning device based on UAV remote sensing includes: An image input module is used to obtain remote sensing image information based on a UAV, wherein the remote sensing image information includes a remote sensing image center and an original remote sensing image, and extract features from the original remote sensing image to obtain remote sensing image features; An image acquisition module, used to obtain vehicle body image information and vehicle body radar information, and obtain overhead image features based on the vehicle body image information and the vehicle body radar information; A remote sensing processing module, configured to perform semantic segmentation on the remote sensing image features to obtain a remote sensing semantic segmentation image, perform contour recognition on the remote sensing image features to obtain a remote sensing contour recognition image, and obtain remote sensing geometric features based on the remote sensing semantic segmentation image and the remote sensing contour recognition image; A bird's-eye view processing module, configured to perform semantic segmentation on the bird's-eye view image features to obtain a bird's-eye view semantic segmentation image, cluster the bird's-eye view semantic segmentation image based on semantic classification and pixel distance to obtain key pixel clusters, and fit the key pixel clusters to obtain bird's-eye view geometric features; The vehicle positioning module is used to obtain the relative coordinate position of the vehicle relative to the center of the remote sensing image based on the matching relationship between the remote sensing geometric features and the overhead geometric features, and to position the vehicle according to the relative coordinate position.
10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the vehicle positioning method based on drone remote sensing as described in any one of claims 1-7.
Citation Information
Cited By
Obstacle detection method and device, electronic equipment and storage medium
CN119068462A
