Vehicle positioning method and device, storage medium and electronic device

Abstract

The invention provides a vehicle positioning method and device, a storage medium and an electronic device, and the method comprises the steps: generating an aerial view of a target vehicle; semantic segmentation is carried out on the aerial view to obtain an aerial view semantic segmentation map, and the aerial view semantic segmentation map comprises category information of each pixel in the aerial view; performing distance transformation on category information of each pixel in the aerial view semantic segmentation map to obtain a distance transformation result; mapping the distance transformation result to category information to obtain a smooth semantic segmentation result of the aerial view semantic segmentation map; registering the aerial view semantic segmentation maps according to a smooth semantic segmentation result, and constructing a similarity function between two adjacent frames of aerial view semantic segmentation maps; according to the similarity function, pose estimation is carried out on the target vehicle in the aerial view semantic segmentation map to obtain a positioning result, and through the vehicle positioning method and device, the technical problem that the calculation complexity is high during vehicle positioning in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle automatic driving technology, and in particular to a vehicle positioning method and device, a storage medium, and an electronic device. Background Art

[0002] A very important application direction of autonomous driving technology is L4 level automatic valet parking technology (Auto Valet Parking), which is to complete fully autonomous driving in a limited place (underground parking lot); a very important technical component of AVP technology is the mapping of this limited place and the positioning of the vehicle body, that is, using the on-board sensors to create a map of the relevant place and estimate the body posture when the vehicle re-enters the place based on the established map.

[0003] Related technologies generally require feature point / semantic vector extraction from images first. This will increase noise (feature point extraction will introduce noise). Second, using only feature points for subsequent calculations will reduce the amount of information used, resulting in inaccurate positioning accuracy. Third, after extracting feature points, we generally need to match them between frames for data association, which will introduce more complex matching rules and unnecessary noise, increase the complexity of calculations, and reduce the clarity of the image.

[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0005] The present invention provides a vehicle positioning method and device, a storage medium, and an electronic device to solve the above technical problems existing in the related art.

[0006] According to one embodiment of the present invention, a vehicle positioning method is provided, characterized in that it includes: generating a bird's-eye view of a target vehicle; performing semantic segmentation on the bird's-eye view to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view; performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation map to obtain a distance transformation result; mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map; registering the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result to construct a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps; and performing vehicle pose estimation on the target vehicle in the bird's-eye view semantic segmentation map according to the similarity function to obtain a positioning result.

[0007] Optionally, generating a bird's-eye view of the target vehicle includes: obtaining surround-view pictures collected by a surround-view camera from multiple perspectives of the target vehicle; using the camera intrinsic parameters of the surround-view camera to perform a dedistortion operation on the surround-view pictures to obtain a first intermediate picture; using the camera extrinsic parameters of the surround-view camera to perform a back-projection transformation on the first intermediate picture, and splicing them to generate a bird's-eye view of the target vehicle.

[0008] Optionally, semantic segmentation is performed on the bird's-eye view to obtain a bird's-eye view semantic segmentation map, including: semantic segmentation is performed on each pixel in the bird's-eye view to obtain a discrete probability that the corresponding pixel belongs to each traffic category, wherein the traffic categories include: lane lines, garage locations, and arrows; after all pixels in the bird's-eye view are semantically segmented, the bird's-eye view semantic segmentation map and the discrete probability distribution of the category to which each pixel in the bird's-eye view belongs are output.

[0009] Optionally, a distance transformation is performed on the category information of each pixel in the bird's-eye view to obtain a distance transformation result, including: for each pixel in the bird's-eye view, determining the first pixel position information of the corresponding pixel; searching for the second pixel position information of the pixel point in the non-background area closest to the first pixel position based on a neighborhood range, wherein the bird's-eye view includes a background area and a non-background area; calculating the distance transformation value of the corresponding pixel according to the first pixel position information and the second pixel position information; after all pixels in the bird's-eye view are transformed, normalizing the distance change values ​​of all pixels in the bird's-eye view to obtain the distance transformation result.

[0010] Optionally, the distance transformation result is mapped to the category information to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map, including: for each pixel point of the distance transformation result, mapping the distance transformation value of the pixel point with the probability distribution of the corresponding pixel point in the bird's-eye view to obtain the semantic distance of each pixel point, wherein the semantic distance is used to characterize the distance between the corresponding pixel point and the traffic category; and integrating the semantic distances of all the pixel points to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map.

[0011] Optionally, the bird's-eye view semantic segmentation map is aligned according to the smoothed semantic segmentation result, and a similarity function between two adjacent bird's-eye view semantic segmentation maps is constructed, including: determining the semantic distance of all corresponding pixel points of two adjacent bird's-eye view semantic segmentation maps; calculating the cross entropy of the semantic distance of all pixel points of the two adjacent bird's-eye view semantic segmentation maps according to the semantic distance; and constructing the similarity function of the two adjacent bird's-eye view semantic segmentation maps according to the sum of the cross entropies of all pixel points.

[0012] Optionally, estimating the posture of the target vehicle in the bird's-eye view semantic segmentation map according to the similarity function to obtain a positioning result includes: performing nonlinear optimization on the similarity function based on a preset value to obtain a maximum value of the similarity function; determining the maximum value as a posture transformation result of the bird's-eye view semantic segmentation map; fusing the posture transformation result with other positioning information to obtain a vehicle positioning result of the target vehicle, wherein the other positioning information is positioning information obtained using sensors other than a camera.

[0013] According to another embodiment of the present invention, a vehicle positioning device is provided, comprising: a generation module for generating a bird's-eye view of a target vehicle; a segmentation module for performing semantic segmentation on the bird's-eye view to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view; a transformation module for performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation map to obtain a distance transformation result; a mapping module for mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map; a registration module for performing registration on the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result to construct a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps; and a positioning module for performing pose estimation on the target vehicle in the bird's-eye view semantic segmentation map according to the similarity function to obtain a positioning result.

[0014] Optionally, the generation module includes: an acquisition unit, used to acquire surround-view images captured by the surround-view camera at multiple perspectives of the target vehicle; a processing unit, used to perform a dedistortion operation on the surround-view images using the camera intrinsic parameters of the surround-view camera to obtain a first intermediate image; and a transformation unit, used to perform a back-projection transformation on the first intermediate image using the camera extrinsic parameters of the surround-view camera, and splice and generate a bird's-eye view of the target vehicle.

[0015] Optionally, the segmentation module includes: a segmentation unit, used to perform semantic segmentation on each pixel in the bird's-eye view to obtain a discrete probability that the corresponding pixel belongs to each traffic category, wherein the traffic categories include: lane lines, garage locations, and arrows; an output unit, used to output a bird's-eye view semantic segmentation map and a discrete probability distribution of the category to which each pixel in the bird's-eye view belongs after all pixels in the bird's-eye view are semantically segmented.

[0016] Optionally, the transformation module includes: a determination unit, used to determine the first pixel position information of the corresponding pixel for each pixel in the bird's-eye view; a search unit, used to search for the second pixel position information of the pixel point in the non-background area closest to the first pixel position based on a neighborhood range, wherein the bird's-eye view includes a background area and a non-background area; a calculation unit, used to calculate the distance transformation value of the corresponding pixel based on the first pixel position information and the second pixel position information; and a processing unit, used to normalize the distance change values ​​of all pixels in the bird's-eye view to obtain the distance transformation result after all pixels in the bird's-eye view are transformed.

[0017] Optionally, the mapping module includes: a mapping unit, used to map the distance transformation value of each pixel point of the distance transformation result with the probability distribution of the corresponding pixel point in the bird's-eye view to obtain the semantic distance of each pixel point, wherein the semantic distance is used to characterize the distance between the corresponding pixel point and the traffic category; an integration unit, used to integrate the semantic distances of all the pixel points to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map.

[0018] Optionally, the registration module includes: a determination unit, used to determine the semantic distance of all corresponding pixel points of two adjacent frames of bird's-eye view semantic segmentation images; a calculation unit, used to calculate the cross entropy of the semantic distance of all pixel points of the two adjacent frames of bird's-eye view semantic segmentation images according to the semantic distance; and a construction unit, used to construct a similarity function of the two adjacent frames of bird's-eye view semantic segmentation images according to the sum of the cross entropies of all pixel points.

[0019] Optionally, the positioning module includes: an optimization unit, used to perform nonlinear optimization on the similarity function based on a preset value to obtain the maximum value of the similarity function; a determination unit, used to determine the maximum value as the posture transformation result of the bird's-eye view semantic segmentation map; a positioning unit, used to fuse the posture transformation result with other positioning information to obtain the vehicle positioning result of the target vehicle, wherein the other positioning information is positioning information obtained using sensors other than the camera.

[0020] According to yet another embodiment of the present invention, a storage medium is provided, wherein a computer program is stored in the storage medium, wherein the computer program is configured to execute the steps of any one of the above-mentioned device embodiments when running.

[0021] According to another embodiment of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned device embodiments.

[0022] Beneficial effects of the present invention:

[0023] 1. The present invention improves the utilization rate of image information, reduces the noise of the image and the complexity of calculation by introducing the semantic distribution of pixels, solves the technical problem of high calculation complexity in related technologies during vehicle positioning, reduces the noise of the image and the complexity of calculation, and improves the accuracy and reliability of vehicle body posture estimation.

[0024] 2. The present invention avoids the computational complexity caused by using data association by directly calculating the similarity between two frames of bird's-eye view semantic segmentation images, reduces the image noise and computational complexity, and thus improves the accuracy and reliability of vehicle body posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;

[0027] Figure 2 is a flow chart of a vehicle positioning method according to an embodiment of the present invention;

[0028] Figure 3 is an overall flow chart of vehicle positioning according to an embodiment of the present invention;

[0029] Figure 4 is a flow chart of posture estimation for vehicle positioning according to an embodiment of the present invention;

[0030] Figure 5 It is a structural block diagram of a vehicle positioning device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Example 1

[0034] The method embodiment provided in the first embodiment of the present application can be executed in a vehicle-mounted computer, a vehicle or a similar computing device. Taking running on a vehicle-mounted computer as an example, Figure 1 FIG. 1 is a hardware structure diagram of a vehicle-mounted computer according to an embodiment of the present invention. Figure 1 As shown, the vehicle-mounted computer may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the vehicle-mounted computer may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] The memory 104 can be used to store on-board computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a vehicle positioning method in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the vehicle-mounted computer server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] In this embodiment, a vehicle positioning method is provided. Figure 2 is a flow chart of a vehicle positioning method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0038] Step S202 generates a bird's-eye view of the target vehicle;

[0039] Optionally, generating a bird's-eye view of the target vehicle requires collecting surround-view photos from multiple perspectives through the surround-view camera of the target vehicle, and then stitching the surround-view photos into a bird's-eye view by methods such as dedistortion and inverse projection.

[0040] Step S204, performing semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view image;

[0041] Optionally, semantic segmentation is called full-pixel semantic segmentation, which is the process of classifying each pixel of the image into a certain semantic area block in a certain way. The semantic segmentation operation in this scheme is to classify each pixel in the bird's-eye view into each traffic category, and the traffic categories include: lane lines, garage locations, and arrows.

[0042] Step S206, performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation image to obtain a distance transformation result;

[0043] Optionally, the category information of each pixel is a discrete probability distribution p(u,v) of the traffic category to which each pixel belongs, where u and v are indexes of the pixel positions, and the distance transformation is to find the non-background pixel point closest to the pixel point within the pixel point domain, calculate the Euclidean distance between the two points, and the distance transformation result obtained is used to characterize the distance from each pixel point to the nearest traffic category.

[0044] Step S208, mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map;

[0045] Optionally, the mapping process is to multiply the distance transformation value corresponding to the discrete probability distribution of each pixel in the semantic segmentation result to obtain the semantic distance value of the pixel point.

[0046] Step S210, registering the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result, and constructing a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps;

[0047] Step S212, estimating the position and posture of the target vehicle in the bird's-eye view semantic segmentation image according to the similarity function to obtain a positioning result.

[0048] Optionally, the positioning data obtained needs to be fused with other vehicle-mounted information sources, such as vehicle-mounted distance sensors, GPS, inertial measurement units (IMU), etc.

[0049] According to the embodiment of the present invention, a bird's-eye view of a target vehicle is generated; semantic segmentation is performed on the bird's-eye view to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view; distance transformation is performed on the category information of each pixel in the bird's-eye view semantic segmentation map to obtain a distance transformation result; the distance transformation result is mapped to the category information to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map; the bird's-eye view semantic segmentation map is aligned according to the smooth semantic segmentation result to construct a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps; the vehicle posture of the target vehicle in the bird's-eye view semantic segmentation map is estimated according to the similarity function to obtain a positioning result, and the utilization rate of image information is improved by introducing the semantic distribution of pixels, and the computational complexity caused by using data association is avoided by directly calculating the similarity between two frames of bird's-eye view semantic segmentation maps, and the noise and computational complexity of the picture are reduced, thereby solving the technical problem of high computational complexity in vehicle positioning in the related technology, reducing the noise and computational complexity of the picture, and improving the accuracy and reliability of vehicle body posture estimation.

[0050] In this embodiment, generating a bird's-eye view of the target vehicle includes: obtaining surround-view images captured by a surround-view camera at multiple perspectives of the target vehicle; using the camera intrinsic parameters of the surround-view camera to perform a dedistortion operation on the surround-view images to obtain a first intermediate image; using the camera extrinsic parameters of the surround-view camera to perform a back-projection transformation on the first intermediate image, and splicing them to generate a bird's-eye view of the target vehicle.

[0051] Optionally, dedistortion is used to correct lens distortion in the image so that objects in the image appear to be more in line with their actual shapes. In this solution, it refers to correcting the surround-view images captured by the surround-view camera to eliminate distortion caused by lens distortion. The inverse projection transformation is used to convert an image from one perspective to another. In this solution, it refers to using the camera intrinsic parameters and camera extrinsic parameters of the surround-view camera to project the dedistorted image from multiple perspectives of the vehicle to a unified perspective, thereby generating a bird's-eye view of the target vehicle.

[0052] In this embodiment, semantic segmentation is performed on the bird's-eye view to obtain a bird's-eye view semantic segmentation map, including: semantic segmentation is performed on each pixel in the bird's-eye view to obtain a discrete probability that the corresponding pixel belongs to each traffic category, wherein the traffic categories include: lane lines, garage locations, and arrows; after all pixels in the bird's-eye view are semantically segmented, the bird's-eye view semantic segmentation map and the discrete probability distribution of the category to which each pixel in the bird's-eye view belongs are output.

[0053] Optionally, semantic segmentation is performed on each pixel in the bird's-eye view, i.e., each pixel in the bird's-eye view is marked as belonging to a certain category in the traffic category, such as lane lines, garage locations, arrows, etc.; the discrete probability of the corresponding pixel belonging to each traffic category is obtained, i.e., for each pixel, the probability distribution of its belonging to each traffic category is calculated, and these probability values ​​represent the possibility of the pixel belonging to different traffic categories; the bird's-eye view semantic segmentation map is an image of the same size as the bird's-eye view, in which each pixel is marked as a specific traffic category.

[0054] In this embodiment, distance transformation is performed on the category information of each pixel in the bird's-eye view image to obtain a distance transformation result, including:

[0055] S11, for each pixel in the bird's-eye view, determining first pixel position information of the corresponding pixel;

[0056] S12, searching for second pixel position information of a pixel point in a non-background area closest to the first pixel position based on a neighborhood range, wherein the bird's-eye view includes a background area and a non-background area;

[0057] Optionally, the bird's-eye view includes a background area and a non-background area, the background area refers to an area without traffic category information, the non-background area refers to an area containing traffic category information, and the pixels in the non-background area are non-background pixels.

[0058] S13, calculating a distance transformation value of a corresponding pixel according to the first pixel position information and the second pixel position information;

[0059] In an example, the distance transformation value of the corresponding pixel is defined as d(x), and the calculation formula of d(x) is d(x)=min(dist(x,y)), where x is the corresponding pixel point, y is the pixel point within the domain of x, the dist(x,y) function is used to calculate the Euclidean distance between the x pixel point and the y pixel point, and the min() function is used to characterize the non-background pixel point that is closest to the x pixel point within the domain of the x pixel point.

[0060] S14, after all pixels of the bird's-eye view are transformed, normalize the distance change values ​​of all pixels of the bird's-eye view to obtain the distance transformation result.

[0061] Optionally, "normalization" refers to linearly transforming a set of values ​​so that they meet certain standards or ranges, usually scaling the values ​​to a specific interval for easier comparison or processing. Specifically, in the previous steps of this solution, the pixels of the bird's-eye view are transformed by distance, resulting in changes in the distance values ​​between pixels. When the transformation of all pixels is completed, the distance transformation values ​​of these pixels need to be normalized, for example, these distance values ​​are linearly mapped so that they are within a specific range, such as [0,1] or [-1,1]. The purpose of normalization is to make the distance transformation values ​​of all pixels have a unified scale, which helps to facilitate further analysis, processing or visualization. The normalization of the distance change values ​​of all pixels in the bird's-eye view, that is, the normalization of the bird's-eye view, does not change the substantial information of the pixels in the bird's-eye view.

[0062] In this embodiment, the distance transformation result is mapped to the category information to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map, including: for each pixel point of the distance transformation result, the distance transformation value of the pixel point is mapped with the probability distribution of the corresponding pixel point in the bird's-eye view to obtain the semantic distance of each pixel point, wherein the semantic distance is used to characterize the distance between the corresponding pixel point and the traffic category; the semantic distances of all the pixels are integrated to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map.

[0063] After calculating the semantic distance of each pixel, the nearest target traffic category can be selected. For example, if pixel a is closest to the lane line, the pixel is attributed to the pixel where the lane line is located, and the interfering pixels are eliminated.

[0064] In an example, the semantic distance of a pixel is defined as sd(x), and the semantic distance of a pixel is sd(x)=d(x)*p(x), where d(x) is the distance transformation value of the pixel, and p(x) is the probability distribution of the pixel.

[0065] Smooth semantic segmentation results belong to data enhancement of semantic segmentation results. Through data enhancement, smoother and more coherent image segmentation can be obtained, making the segmentation boundaries more natural and clear, so as to obtain more accurate vehicle positioning results.

[0066] In this embodiment, the bird's-eye view semantic segmentation map is registered according to the smoothed semantic segmentation result, and a similarity function between two adjacent bird's-eye view semantic segmentation maps is constructed, including:

[0067] S21, determining the semantic distances of all corresponding pixels of two adjacent frames of bird's-eye view semantic segmentation images;

[0068] Optionally, the pixel information of the two images is stored in an array, and the pixel information in the array is read in a loop to achieve a one-to-one correspondence between all the pixels of two adjacent frames of the bird's-eye view semantic segmentation image.

[0069] S22, calculating the cross entropy of the semantic distances of all pixel points of the bird's-eye view semantic segmentation images of two adjacent frames according to the semantic distances;

[0070] In an example, the cross entropy calculation formula for the semantic distance between corresponding pixel points of two adjacent bird's-eye view semantic segmentation images is sim(sd(x1), sd(x2)) = cross_entropy(sd(x1), sd(x2)), where cross_entropy is the cross entropy of the loss function and sim is the similarity function value of a single corresponding pixel point.

[0071] S23, constructing a similarity function of the two adjacent bird's-eye view semantic segmentation images according to the sum of the cross entropies of all pixels.

[0072] Optionally, the similarity function J(T)=sim(SDT(I1), SDT(I2)) of the pose transformation is obtained by summing the cross entropy of all pixels of the two frames of bird's-eye view semantic segmentation images, where SDT(I) represents the smoothed semantic segmentation result of the bird's-eye view semantic segmentation image of the I-th frame, that is, the sum of the distance transformation values ​​of all pixels of the bird's-eye view semantic segmentation image of the I-th frame.

[0073] In this embodiment, estimating the posture of the target vehicle in the bird's-eye view semantic segmentation map according to the similarity function to obtain a positioning result includes: performing nonlinear optimization on the similarity function based on a preset value to obtain a maximum value of the similarity function; determining the maximum value as a posture transformation result of the bird's-eye view semantic segmentation map; fusing the posture transformation result with other positioning information to obtain a vehicle positioning result of the target vehicle, wherein the other positioning information is positioning information obtained using sensors other than a camera.

[0074] Optionally, "non-linear optimization" refers to an optimization technique for finding variable values ​​that can make a specific function reach a maximum or minimum value, but this function is not necessarily linear, and may be a complex, non-linear function. The function in this solution is a similarity function. Finding the parameter value that makes the similarity function reach the maximum value can be achieved through iteration and other methods. Other positioning information comes from sensors other than the camera, such as GPS, inertial measurement unit, etc.

[0075] Figure 3 is an overall flow chart of vehicle positioning according to an embodiment of the present invention, such as Figure 3 The process shown includes the following steps:

[0076] S31, surround image dedistortion;

[0077] Optionally, the images acquired by the four surround-view cameras are dedistorted according to the camera intrinsic parameters. Dedistortion is used to correct the lens distortion in the image so that the objects in the image appear to be more consistent with the actual shape. In this solution, it refers to correcting the surround-view images acquired by the surround-view cameras to eliminate the distortion caused by lens distortion.

[0078] S32, inverse projection transformation + splicing;

[0079] Optionally, the dedistorted image is inversely projected using the camera's external parameters, the image is projected to a bird's-eye view, and then image stitching is performed to obtain a bird's-eye view stitched from the perspectives of four surround cameras.

[0080] S33, semantic segmentation;

[0081] Semantic segmentation is performed on the bird's-eye view, and the output is the category distribution of each pixel. The categories are generally lane lines, garage locations, arrows and other traffic signs. Compared with visual feature points, they are more specific and persistent.

[0082] S34, pose estimation;

[0083] Figure 4 is a flow chart of posture estimation for vehicle positioning according to an embodiment of the present invention, such as Figure 4 The process shown includes the following steps:

[0084] Distance transformation of classification, input the result of semantic segmentation, that is, the discrete probability distribution p(u,v) of the category to which each pixel belongs, where u and v are the indexes of the pixel position. First calculate the distance transformation (DistanceTransform) at the pixel level, as defined by the following formula: d(x) = min(dist(x,y)), y belongs to a certain range of the neighborhood of x; where x and y are pixels in the image, and dist(x,y) is the Euclidean distance between the two points; the physical meaning of the above formula is that the distance transformation value of each pixel is the Euclidean distance of this pixel to the nearest non-background pixel; then normalize the obtained transformation image; obtain the distance transformation DT(I) of the semantic segmentation map;

[0085] Semantic mapping uses the original probability distribution p(x) of each pixel (from the result of semantic segmentation) to map the result of distance transformation to the distribution of each category. It is defined as follows: sd(x) = d(x)*p(x), where sd(x) is the semantic distance of a pixel. It can be understood that a smooth semantic segmentation result SDT(I) of a bird's-eye view is actually obtained in the end.

[0086] Image registration, for each frame of the original semantic segmentation result, the above two steps are performed to obtain a smooth semantic segmentation result SDT(I). In the image registration step, an attempt will be made to construct a similarity function J = sim(SDT(I1), SDT(I2)) between the two frames of bird's-eye view semantic segmentation images I1 and I2, using the category probability function of each pixel to describe the similarity between the two frames of images, which is defined as follows: sim(sd(x1), sd(x2)) = cross_entropy(sd(x1), sd(x2));

[0087] Pose estimation, according to the similarity function J = sim (SDT (I1), SDT (I2)), J is the sum of the similarities of all overlapping pixels; based on a suitable initial value, the pose transformation T that maximizes the objective function is obtained with the help of nonlinear optimization technology best ; The pose transformation obtained using the semantic segmentation image can be fused with other positioning information sources to obtain more accurate and robust positioning results.

[0088] In this embodiment, a surround view image collected by a surround view camera of the vehicle is subjected to a dedistortion operation, an inverse projection transformation operation and then spliced ​​to generate a bird's-eye view of the target vehicle, and then the bird's-eye view image is semantically segmented to obtain a bird's-eye view semantic segmentation map, and the category information of each pixel in the bird's-eye view semantic segmentation map is distance transformed to obtain a distance transformation result; the distance transformation result is mapped to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map; the cross entropy of the semantic distances of all pixels between two adjacent frames of the bird's-eye view semantic segmentation map is calculated based on the semantic distance value of the smoothed semantic segmentation result, and the semantic distances of all pixels between the two adjacent frames of the bird's-eye view semantic segmentation map are calculated based on the semantic distance value of the smoothed semantic segmentation result. The sum of the cross entropy of all pixel points constructs the similarity function of the two adjacent frames of the bird's-eye view semantic segmentation map; according to the similarity function, the vehicle posture of the target vehicle in the bird's-eye view semantic segmentation map is estimated to obtain a positioning result, and by using all pixel points for pixel point extraction, the technical problem that the related technology extracts feature points from the picture and introduces noise, making the picture unclear, is solved. By directly calculating the similarity between the two frames of the bird's-eye view semantic segmentation map, the computational complexity caused by using data association is avoided, the noise of the picture and the complexity of the calculation are reduced, thereby improving the accuracy and reliability of the vehicle posture estimation.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0090] Example 2

[0091] In the present embodiment, a vehicle positioning device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and hardware for the reservation function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, can also be conceived.

[0092] Figure 5 is a structural block diagram of a vehicle positioning device according to an embodiment of the present invention, such as Figure 5 As shown, the device comprises:

[0093] A generating module 50, for generating a bird's-eye view of a target vehicle;

[0094] A segmentation module 51 is used to perform semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view image;

[0095] A transformation module 52, configured to perform distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation image to obtain a distance transformation result;

[0096] A mapping module 53, used for mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map;

[0097] A registration module 54, configured to register the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result, and construct a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps;

[0098] The positioning module 55 is used to estimate the position and posture of the target vehicle in the bird's-eye view semantic segmentation image according to the similarity function to obtain a positioning result.

[0099] Optionally, the generation module includes: an acquisition unit, used to acquire surround-view images captured by the surround-view camera at multiple perspectives of the target vehicle; a processing unit, used to perform a dedistortion operation on the surround-view images using the camera intrinsic parameters of the surround-view camera to obtain a first intermediate image; and a transformation unit, used to perform a back-projection transformation on the first intermediate image using the camera extrinsic parameters of the surround-view camera, and splice and generate a bird's-eye view of the target vehicle.

[0100] Optionally, the segmentation module includes: a segmentation unit, used to perform semantic segmentation on each pixel in the bird's-eye view to obtain a discrete probability that the corresponding pixel belongs to each traffic category, wherein the traffic categories include: lane lines, garage locations, and arrows; an output unit, used to output a bird's-eye view semantic segmentation map and a discrete probability distribution of the category to which each pixel in the bird's-eye view belongs after all pixels in the bird's-eye view are semantically segmented.

[0101] Optionally, the transformation module includes: a determination unit, used to determine the first pixel position information of the corresponding pixel for each pixel in the bird's-eye view; a search unit, used to search for the second pixel position information of the pixel point in the non-background area closest to the first pixel position based on a neighborhood range, wherein the bird's-eye view includes a background area and a non-background area; a calculation unit, used to calculate the distance transformation value of the corresponding pixel based on the first pixel position information and the second pixel position information; and a processing unit, used to normalize the distance change values ​​of all pixels in the bird's-eye view to obtain the distance transformation result after all pixels in the bird's-eye view are transformed.

[0102] Optionally, the mapping module includes: a mapping unit, used to map the distance transformation value of each pixel point of the distance transformation result with the probability distribution of the corresponding pixel point in the bird's-eye view to obtain the semantic distance of each pixel point, wherein the semantic distance is used to characterize the distance between the corresponding pixel point and the traffic category; an integration unit, used to integrate the semantic distances of all the pixel points to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map.

[0103] Optionally, the registration module includes: a determination unit, used to determine the semantic distance of all corresponding pixel points of two adjacent frames of bird's-eye view semantic segmentation images; a calculation unit, used to calculate the cross entropy of the semantic distance of all pixel points of the two adjacent frames of bird's-eye view semantic segmentation images according to the semantic distance; and a construction unit, used to construct a similarity function of the two adjacent frames of bird's-eye view semantic segmentation images according to the sum of the cross entropies of all pixel points.

[0104] Optionally, the positioning module includes: an optimization unit, used to perform nonlinear optimization on the similarity function based on a preset value to obtain the maximum value of the similarity function; a determination unit, used to determine the maximum value as the posture transformation result of the bird's-eye view semantic segmentation map; a positioning unit, used to fuse the posture transformation result with other positioning information to obtain the vehicle positioning result of the target vehicle, wherein the other positioning information is positioning information obtained using sensors other than the camera.

[0105] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0106] Example 3

[0107] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0108] Optionally, in this embodiment, the above storage medium may be configured to store a computer program for executing:

[0109] S1, generate a bird’s-eye view of the target vehicle;

[0110] S2, performing semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation map, where the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view image;

[0111] S3, performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation image to obtain a distance transformation result;

[0112] S4, mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map;

[0113] S5, registering the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result, and constructing a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps;

[0114] S6, estimating the position and posture of the target vehicle in the bird's-eye view semantic segmentation image according to the similarity function to obtain a positioning result.

[0115] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0116] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0117] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0118] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0119] S1, generate a bird’s-eye view of the target vehicle;

[0120] S2, performing semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation map, where the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view image;

[0121] S3, performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation image to obtain a distance transformation result;

[0122] S4, mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map;

[0123] S5, registering the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result, and constructing a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps;

[0124] S6, estimating the position and posture of the target vehicle in the bird's-eye view semantic segmentation image according to the similarity function to obtain a positioning result.

[0125] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0126] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0127] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0132] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A vehicle positioning method, characterized in that: include: Generate a bird's-eye view of the target vehicle; Performing semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view image; Performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation image to obtain a distance transformation result; Mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map; The bird's-eye view semantic segmentation map is registered according to the smoothed semantic segmentation result, and a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps is constructed; The position and posture of the target vehicle in the bird's-eye view semantic segmentation image is estimated according to the similarity function to obtain a positioning result.

2. The method according to claim 1, characterized in that Generating a bird's-eye view of the target vehicle includes: Acquire surround view images collected by a surround view camera at multiple viewing angles of the target vehicle; Performing a dedistortion operation on the surround view image using the camera intrinsic parameters of the surround view camera to obtain a first intermediate image; The first intermediate image is subjected to inverse projection transformation using camera extrinsics of the surround view camera, and is spliced ​​to generate a bird's-eye view of the target vehicle.

3. The method according to claim 1, characterized in that Performing semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation image, including: Performing semantic segmentation on each pixel in the bird's-eye view image to obtain a discrete probability that the corresponding pixel belongs to each traffic category, wherein the traffic categories include: lane lines, garage locations, and arrows; After all pixels in the bird's-eye view image are semantically segmented, a bird's-eye view semantic segmentation image and a discrete probability distribution of the category to which each pixel in the bird's-eye view image belongs are output.

4. The method according to claim 1, characterized in that: Performing distance transformation on the category information of each pixel in the bird's-eye view image to obtain a distance transformation result includes: For each pixel in the bird's-eye view, determining first pixel position information of the corresponding pixel; Searching for second pixel position information of a pixel point in a non-background area closest to the first pixel position based on a neighborhood range, wherein the bird's-eye view includes a background area and a non-background area; Calculating a distance transformation value of a corresponding pixel according to the first pixel position information and the second pixel position information; After all pixels of the bird's-eye view image are transformed, the distance change values ​​of all pixels of the bird's-eye view image are normalized to obtain the distance transformation result.

5. The method according to claim 4, characterized in that Mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map includes: For each pixel point of the distance transformation result, the distance transformation value of the pixel point is mapped with the probability distribution of the corresponding pixel point in the bird's-eye view to obtain the semantic distance of each pixel point, wherein the semantic distance is used to characterize the distance between the corresponding pixel point and the traffic category; The semantic distances of all the pixel points are integrated to obtain a smooth semantic segmentation result of the bird's-eye view semantic segmentation map.

6. The method according to claim 1, characterized in that The bird's-eye view semantic segmentation map is registered according to the smoothed semantic segmentation result, and a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps is constructed, including: Determine the semantic distances of all corresponding pixels of two adjacent frames of bird's-eye view semantic segmentation images; Calculate the cross entropy of the semantic distances of all pixel points in the bird's-eye view semantic segmentation images of two adjacent frames according to the semantic distance; The similarity function of the bird's-eye view semantic segmentation images of the two adjacent frames is constructed according to the sum of the cross entropies of all pixels.

7. The method according to claim 1, characterized in that Estimating the position and posture of the target vehicle in the bird's-eye view semantic segmentation image according to the similarity function to obtain a positioning result includes: Performing nonlinear optimization on the similarity function based on a preset value to obtain a maximum value of the similarity function; Determine the maximum value as the pose transformation result of the bird's-eye view semantic segmentation map; The posture transformation result is integrated with other positioning information to obtain the vehicle positioning result of the target vehicle, wherein the other positioning information is positioning information obtained by using other sensors except the camera.

8. A vehicle positioning device based on surround semantic segmentation splicing graph, characterized in that: include: A generation module, used to generate a bird's-eye view of the target vehicle; A segmentation module is used to perform semantic segmentation on the bird's-eye view image to obtain a bird's-eye view semantic segmentation map, wherein the bird's-eye view semantic segmentation map includes category information of each pixel in the bird's-eye view image; A transformation module, used for performing distance transformation on the category information of each pixel in the bird's-eye view semantic segmentation image to obtain a distance transformation result; A mapping module, used for mapping the distance transformation result to the category information to obtain a smoothed semantic segmentation result of the bird's-eye view semantic segmentation map; A registration module, used for registering the bird's-eye view semantic segmentation map according to the smoothed semantic segmentation result, and constructing a similarity function between two adjacent frames of bird's-eye view semantic segmentation maps; A positioning module is used to estimate the position and posture of the target vehicle in the bird's-eye view semantic segmentation image according to the similarity function to obtain a positioning result.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images