Parking lot three-dimensional reconstruction method, device and equipment and storage medium

Abstract

The invention relates to the technical field of automatic driving, and discloses a parking lot three-dimensional reconstruction method and device, equipment and a storage medium, and the method comprises the steps: obtaining a plurality of key frame image pairs corresponding to a target parking scene; extracting a plurality of dense matching point pairs of each key frame image pair, and obtaining confidence and point pair positions of each dense matching point pair; obtaining matching point pairs of which the confidence is greater than or equal to the target confidence and the point pair positions are target positions in the dense matching point pairs to obtain a plurality of candidate matching point pairs; calculating the kernel density of the plurality of candidate matching point pairs, and adjusting the candidate matching point pairs based on the kernel density to obtain target matching point pairs; and based on the target matching point pair, performing three-dimensional reconstruction on the target parking lot scene to obtain a parking lot three-dimensional reconstruction result of the target parking lot scene. The method can improve the precision and robustness of three-dimensional reconstruction of the parking lot.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a parking lot three-dimensional reconstruction method, device, equipment and storage medium. Background Art

[0002] Automatic parking means that the vehicle can park itself in a parking space without manual control by the driver. The application of this technology enables the driver to easily park the vehicle accurately in the designated location. In the automatic parking scenario, the parking environment of the parking lot needs to be reconstructed in three dimensions. However, when the relevant technology reconstructs the parking scene of the parking lot in three dimensions, there is a problem that the reconstruction result is inconsistent with the actual scene. Summary of the invention

[0003] In view of this, the present invention provides a parking lot three-dimensional reconstruction method, device, equipment and storage medium to solve the problem that when relevant technicians perform three-dimensional reconstruction of the parking scene of the parking lot, the reconstruction result is inconsistent with the actual scene.

[0004] In a first aspect, the present invention provides a method for three-dimensional reconstruction of a parking lot, the method comprising:

[0005] Acquire multiple key frame image pairs corresponding to the target parking scene;

[0006] Extract multiple dense matching point pairs of each key frame image pair, and obtain the confidence and point pair position of each dense matching point pair;

[0007] Obtain matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions, so as to obtain multiple candidate matching point pairs;

[0008] Calculate the kernel density of multiple candidate matching point pairs, and adjust the candidate matching point pairs based on the kernel density to obtain the target matching point pairs;

[0009] Based on the target matching point pairs, the target parking scene is three-dimensionally reconstructed to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0010] In some optional implementations, obtaining a plurality of key frame image pairs corresponding to the target parking scene includes:

[0011] Acquire a parking video shot by the vehicle during parking in a target parking scene, where the parking video is shot based on multiple panoramic cameras set on the vehicle;

[0012] Perform frame extraction on the parking video to obtain a parking image set;

[0013] Based on the image overlap between adjacent parking image frames in the parking image set, a plurality of key frame image pairs corresponding to the target parking scene are determined.

[0014] In some optional implementations, extracting multiple dense matching point pairs of each key frame image pair, and obtaining the confidence and point pair position of each dense matching point pair includes:

[0015] Input each key frame image pair into the target encoder for feature extraction to obtain multiple dense matching point pairs and corresponding confidence levels of each key frame image pair;

[0016] Perform semantic segmentation on the key frame image pair to obtain the parking lot area where the key frame image pair is located;

[0017] Based on the parking lot area where the key frame image pair is located, the point pair positions of the densely matched point pairs are determined.

[0018] In some optional implementations, obtaining matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions, to obtain multiple candidate matching point pairs, includes:

[0019] Compare the confidence of each dense matching point pair with the target confidence to obtain a confidence comparison result;

[0020] Based on the confidence comparison results, the matching point pairs with confidence less than the target confidence are eliminated from the dense matching point pairs, and the cleaning matching point pairs with confidence greater than or equal to the target confidence are obtained;

[0021] The point pair positions of the cleaning matching point pairs are obtained, and the cleaning matching point pairs whose point pair positions are target positions are used as candidate matching point pairs, where the target positions are used to represent the positions for parking in the parking lot area.

[0022] In some optional implementations, calculating the kernel density of multiple candidate matching point pairs, and adjusting the candidate matching point pairs based on the kernel density to obtain target matching point pairs includes:

[0023] Performing statistical analysis on multiple candidate matching point pairs to obtain kernel density functions corresponding to the multiple candidate matching point pairs;

[0024] Calculate the kernel density of each candidate matching point pair based on the kernel density function;

[0025] The inverse of the kernel density is used as the weight of the candidate matching point;

[0026] The candidate matching point pairs are adjusted based on the product of the weighted weight and the candidate matching point pairs to obtain the target matching point pairs.

[0027] In some optional implementations, based on the target matching point pairs, a target parking scene is three-dimensionally reconstructed to obtain a three-dimensional reconstruction result of a parking lot of the target parking scene, including:

[0028] Get the camera posture information corresponding to the key frame image;

[0029] Based on the camera posture information and the target matching point pairs, the sparse three-dimensional point cloud data of the target parking scene is obtained;

[0030] The target parking scene is 3D reconstructed based on the sparse 3D point cloud data to obtain a 3D reconstruction result of the parking lot of the target parking scene.

[0031] In some optional implementations, based on the target matching point pairs, the target parking scene is three-dimensionally reconstructed to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene, further comprising:

[0032] Obtain a grid model of the target parking scene;

[0033] Obtain color data and semantic data of corresponding mesh vertices in the mesh model;

[0034] Determine the three-dimensional coordinate data of the mesh vertices based on the target matching point pairs;

[0035] Based on the 3D coordinate data, color data and semantic data, the grid model is rendered to obtain the 3D reconstruction result of the parking lot of the target parking scene.

[0036] In a second aspect, the present invention provides a parking lot three-dimensional reconstruction device, the device comprising:

[0037] An image pair acquisition module, used to acquire multiple key frame image pairs corresponding to a target parking scene;

[0038] A point pair information acquisition module is used to extract multiple dense matching point pairs of each key frame image pair, and obtain the confidence and point pair position of each dense matching point pair;

[0039] A matching point pair acquisition module is used to acquire matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions in the dense matching point pairs, so as to obtain multiple candidate matching point pairs;

[0040] A kernel density calculation module is used to calculate the kernel density of multiple candidate matching point pairs, and adjust the candidate matching point pairs based on the kernel density to obtain target matching point pairs;

[0041] The parking lot reconstruction module is used to perform three-dimensional reconstruction of the target parking scene based on the target matching point pairs to obtain the parking lot three-dimensional reconstruction result of the target parking scene.

[0042] In a third aspect, the present invention provides a computer device, comprising:

[0043] The memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the parking lot three-dimensional reconstruction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the parking lot three-dimensional reconstruction method of the first aspect or any corresponding embodiment thereof.

[0045] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the parking lot three-dimensional reconstruction method of the first aspect or any corresponding embodiment thereof.

[0046] The technical solution provided by the embodiment of the present invention obtains multiple key frame image pairs corresponding to the target parking scene to extract multiple dense matching point pairs of each key frame image pair, and obtains the confidence and point pair position of each dense matching point pair; obtains matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair position is the target position in the dense matching point pairs to obtain multiple candidate matching point pairs, so as to reduce the influence of the inconsistency of edge features on the matching relationship, thereby improving the reliability of subsequent sparse sampling; calculates the kernel density of multiple candidate matching point pairs, and adjusts the candidate matching point pairs based on the kernel density to obtain the target matching point pairs, thereby balancing the distribution of matching points, which can effectively avoid the deviation caused by excessive repeated sampling of high-density areas in the preliminary sampling, and finally generates a set of more comprehensive and evenly distributed balanced samples; by performing three-dimensional reconstruction of the target parking scene based on the target matching point pairs, the reliability of the scene reconstruction is improved, and the situation where the reconstruction result is inconsistent with the actual scene is avoided.

[0047] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a schematic flow chart of a parking lot 3D reconstruction method according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of obtaining candidate matching point pairs in a parking lot 3D reconstruction method according to an embodiment of the present invention;

[0051] Figure 3 is a partial reconstruction effect diagram of a parking scene according to an embodiment of the present invention;

[0052] Figure 4 is a reconstruction effect diagram of an underground driving scene according to an embodiment of the present invention;

[0053] Figure 5 It is a projection effect diagram of the lane line true value and the parking lot three-dimensional reconstruction result of the embodiment of the present invention;

[0054] Figure 6 is another schematic diagram of a process of a parking lot 3D reconstruction method according to an embodiment of the present invention;

[0055] Figure 7 is a structural block diagram of a parking lot 3D reconstruction device according to an embodiment of the present invention;

[0056] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] Figure 1 The flowchart of the first embodiment of the method for three-dimensional reconstruction of a parking lot according to the present invention is shown, which can be used in the above-mentioned computer device. Figure 1 As shown, the method comprises the following steps:

[0059] Step 110: Acquire multiple key frame image pairs corresponding to the target parking scene.

[0060] Among them, the multiple key frame image pairs corresponding to the target parking scene are obtained based on the parking video shot by multiple panoramic cameras set on the vehicle during parking in the target parking scene.

[0061] In some optional embodiments, when obtaining multiple key frame image pairs corresponding to the target parking scene, a parking video shot of the vehicle during parking in the target parking scene can be obtained; the parking video is subjected to frame extraction processing to obtain a parking image set; and based on the image overlap between adjacent frame parking images in the parking image set, the multiple key frame image pairs corresponding to the target parking scene are determined.

[0062] Specifically, obtaining the video captured by the vehicle during parking in the target parking scene is usually achieved by installing multiple panoramic cameras on the vehicle, such as full fisheye cameras (multi-eye fisheye cameras), which can continuously capture images. These cameras may be fixed on different parts of the vehicle, such as the front left, front, front right, and rear left, rear, and rear right, to ensure that the area around the vehicle can be fully covered. During the parking process, the camera records the dynamics and surrounding environment of the vehicle in real time, providing sufficient image data for subsequent three-dimensional reconstruction. When processing the parking video, a parking image set is obtained by extracting frames, and key frames can be selected according to a preset frame rate or according to changes in video content to ensure that the parking image set can accurately reflect the dynamics and environmental layout of the vehicle in the target parking scene. By calculating the image overlap between adjacent frames in the parking image set, multiple key frame image pairs corresponding to the target parking scene can be determined, that is, key frame image pairs with image overlap greater than a target value, thereby obtaining the dynamics and environmental layout of the vehicle in the target parking scene, and providing the necessary image data for three-dimensional reconstruction. In addition, image enhancement and feature extraction can be performed on key frame images to optimize image quality and extract key feature points, thereby improving the accuracy and efficiency of 3D reconstruction.

[0063] Step 120: extract multiple dense matching point pairs of each key frame image pair, and obtain the confidence and point pair position of each dense matching point pair.

[0064] As described above, multiple dense matching point pairs of each key frame image pair are extracted, and the confidence and point pair position of each dense matching point pair are obtained, so as to perform data cleaning on the dense matching point pairs based on the confidence and point pair position.

[0065] In some optional embodiments, when extracting multiple dense matching point pairs of each key frame image pair and obtaining the confidence and point pair position of each dense matching point pair, each key frame image pair can be input into a target encoder for feature extraction to obtain multiple dense matching point pairs and corresponding confidences of each key frame image pair; semantic segmentation is performed on the key frame image pairs to obtain the parking lot area where the key frame image pairs are located; and based on the parking lot area where the key frame image pairs are located, the point pair positions of the dense matching point pairs are determined.

[0066] In practice, see Figure 2First, based on the surround view system (Surround View System) constructed by multiple surround view cameras installed on the vehicle, the parking video shot by the vehicle during parking in the target parking scene is processed by frame extraction to obtain a parking image set, and based on the image overlap between adjacent frames of parking images in the parking image set, multiple key frame image pairs (Image pairs) of the target parking scene are determined. Then, through the encoder (Encoder) combined with the kernel matching and Transformer method, dense matching point pairs are extracted between each key frame image pair, and their confidence is evaluated to obtain multiple dense matching point pairs and corresponding confidence (Dense Warp and Certainty) of each key frame image pair. Subsequently, the key frame image pairs are semantically segmented (Segmentation Block) to obtain the parking area where the key frame image pair is located. Finally, based on the parking lot area where the key frame image pair is located, the point pair positions of the dense matching point pairs are determined to filter out the matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions, i.e., sampling of matching points, to obtain multiple candidate matching point pairs, i.e., coarse matching relationships. Among them, when performing semantic segmentation on the key frame image pairs, the SAM model can be used to accurately extract the road surface area. The segmentation results are used as auxiliary information to shield the matching points in the non-road surface area, thereby reducing the impact of the inconsistency of edge features on the matching relationship, thereby improving the reliability of subsequent sparse sampling.

[0067] Step 130 , obtaining matching point pairs whose confidences are greater than or equal to the target confidences and whose point pair positions are the target positions, to obtain a plurality of candidate matching point pairs.

[0068] The above target confidence is determined through experiments and statistical analysis to ensure that the selected matching point pairs have high accuracy and reliability. The above target positions are positions that can reflect the structural layout of the parking lot in detail, including key positions such as parking spaces, passages, entrances and exits.

[0069] In some optional embodiments, when obtaining matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions to obtain multiple candidate matching point pairs, the confidence of each dense matching point pair can be compared with the target confidence to obtain a confidence comparison result; based on the confidence comparison result, the matching point pairs whose confidence is less than the target confidence in the dense matching point pairs are eliminated to obtain cleaning matching point pairs whose confidence is greater than or equal to the target confidence; the point pair positions of the cleaning matching point pairs are obtained, and the cleaning matching point pairs whose point pair positions are the target positions are used as candidate matching point pairs, and the target positions are used to characterize the positions for parking in the parking lot area, thereby ensuring that the sampling points not only have a high matching confidence, but are also located in a semantically reliable road surface area.

[0070] It can be understood that the screening of the above dense matching relationships is mainly based on the confidence threshold and semantic information, that is, firstly, the matching points with confidence lower than the threshold are eliminated; secondly, only the matching point pairs located in the road surface area are retained, so that the obtained candidate matching point pairs are more targeted. That is, the semantic information of the road surface area is used as a priori input in the form of a mask to directly supervise the sampling process. This supervision mechanism based on semantic masks can guide the model to focus on road surface areas with important geometric significance during the training and inference stages, while avoiding sampling unstable features in the edge areas. This not only improves the targeting of sampling, but also helps the model learn matching relationships more accurately.

[0071] Step 140 , calculating the kernel density of multiple candidate matching point pairs, and adjusting the candidate matching point pairs based on the kernel density to obtain target matching point pairs.

[0072] As described above, the distribution density of each candidate matching point pair is calculated by the kernel density estimation method. Kernel density estimation is a non-parametric estimation method used to estimate the probability density function. It smoothes the data points by placing a kernel function (such as a Gaussian kernel) around each data point to obtain a density estimate of the entire data set. In this embodiment, kernel density estimation is used to evaluate the distribution of candidate matching point pairs in the parking lot area, so as to identify those point pairs that are more concentrated in space and are more likely to represent the actual parking space locations.

[0073] In addition, based on the calculated kernel density, the candidate matching point pairs are adjusted to improve the accuracy of the matching point pairs. Specifically, a kernel density threshold can be set, and only when the kernel density of the candidate matching point pairs is greater than or equal to the threshold can they be considered as target matching point pairs. In this way, through the screening of kernel density, those point pairs with low density, which may be noise or mismatched, can be effectively eliminated, so as to obtain more accurate parking space location information. Finally, these target matching point pairs can be used in the 3D reconstruction algorithm to construct a 3D model of the parking area.

[0074] In some optional implementations, the kernel density of multiple candidate matching point pairs is calculated, and the candidate matching point pairs are adjusted based on the kernel density to obtain the target matching point pairs. It is also possible to perform statistical analysis on the multiple candidate matching point pairs to obtain the kernel density function corresponding to the multiple candidate matching point pairs; calculate the kernel density of each candidate matching point pair based on the kernel density function; use the inverse of the kernel density as the weighted weight of the candidate matching point; adjust the candidate matching point pairs based on the product of the weighted weight and the candidate matching point pairs to obtain the target matching point pairs. It is understandable that since screening based solely on confidence and semantic masks may result in insufficient diversity of matching relationships, this may limit subsequent scene reconstruction and semantic understanding. Therefore, in order to enhance the diversity and coverage of matching relationships, it is necessary to perform balanced sampling of candidate matching point pairs to obtain target matching point pairs.

[0075] In some optional embodiments, a kernel density estimate (KDE) can also be calculated for multiple candidate matching point pairs in a four-dimensional matching space. The four-dimensional matching space is usually composed of matching information of image coordinates and feature space, and the role of KDE is to measure the distribution density of matching points in space. Areas with higher density indicate that the distribution of matching points is concentrated, while areas with lower density indicate sparse distribution. Through kernel density estimation, the distribution density of each candidate matching point pair in space can be evaluated, thereby identifying those point pairs that appear more frequently in dense areas. These point pairs usually have higher confidence because they are consistently matched in multiple perspectives. Next, the results of the kernel density estimation can be used to weight the multiple candidate matching point pairs obtained by preliminary sampling, and give priority to those point pairs with higher kernel density values. Such processing not only retains candidate matching point pairs with high confidence, but also further improves the diversity and representativeness of candidate matching point pairs through kernel density weighting, effectively avoiding the deviation caused by excessive repeated sampling in high-density areas in the initial sampling, and finally generating a set of more comprehensive and evenly distributed balanced samples, that is, obtaining a target matching point pair (coarse matching point pair set) that contains both high-quality matching point pairs and wide coverage, providing a solid foundation for subsequent 3D reconstruction and semantic understanding.

[0076] In some optional implementations, after completing the kernel density estimation and weighting processing, the distance between each candidate matching point pair and its nearest neighbor candidate matching point pair can also be calculated to evaluate the distribution uniformity of the candidate matching point pair in the entire four-dimensional matching space. If the nearest neighbor distance of a candidate matching point pair is significantly smaller than the average distance, it indicates that the candidate matching point pair may be in a locally over-dense area, and therefore has a higher risk of mismatching, and such points will be eliminated.

[0077] Secondly, the quality of candidate matching point pairs can be further verified by matching consistency checks, including checking whether the feature descriptions of candidate matching point pairs under different viewing angles are consistent and whether they satisfy geometric constraints (such as epipolar constraints). Only those points that meet both spatial distribution uniformity and matching consistency requirements will be retained to form the final target matching point pairs, i.e., the balanced sample set.

[0078] It can be understood that the above balanced sampling strategy not only ensures the high confidence and semantic validity of the matching points, but also significantly improves the diversity and coverage of the matching relationships. This provides a more accurate and reliable set of matching points for subsequent BEV perception tasks, such as 3D reconstruction and semantic understanding, thus helping to improve the performance and robustness of the entire system.

[0079] Step 150 , based on the target matching point pairs, three-dimensionally reconstruct the target parking scene to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0080] Specifically, the camera posture information corresponding to the key frame image can be obtained; based on the camera posture information and the target matching point pair, the sparse 3D point cloud data of the target parking scene can be obtained; based on the sparse 3D point cloud data, the target parking scene can be 3D reconstructed to generate a 3D model of the target parking scene. The local reconstruction effect diagram of the ground parking scene is shown in FIG. Figure 3 As shown in the figure, the reconstruction effect of the underground driving scene is as follows Figure 4 As shown in the figure, the 3D model can reflect the structural layout of the parking lot in detail, including parking spaces, passages, entrances and exits and other key elements, thereby achieving accurate measurement and analysis of the parking lot space and providing data support for intelligent parking management. In addition, the 3D model can also be used to simulate and plan the efficiency of parking lot use and optimize the allocation of parking resources.

[0081] In addition, in order to verify the accuracy of the parking lot 3D reconstruction result of the target parking scene, some lane line true value points (i.e., annotation information) can also be obtained and projected onto the reconstruction result to verify the accuracy of the parking lot 3D reconstruction result of the target parking scene. The projection effect diagram of the lane line true value and the parking lot 3D reconstruction result of this embodiment is shown in FIG. Figure 5 From the effect of projecting the lane line true value points onto the reconstruction result in this embodiment, it can be seen that the lane line true value is completely consistent with the reconstruction result.

[0082] The parking lot 3D reconstruction method provided by the embodiment of the present invention obtains multiple key frame image pairs corresponding to the target parking scene to extract multiple dense matching point pairs of each key frame image pair, and obtains the confidence and point pair position of each dense matching point pair; obtains matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair position is the target position in the dense matching point pairs to obtain multiple candidate matching point pairs, thereby significantly reducing the influence of the edge distortion of the fisheye camera on feature extraction and matching, thereby reducing the influence of the inconsistency of edge features on the matching relationship, and improving the reliability of subsequent sparse sampling; calculates the kernel density of multiple candidate matching point pairs, and adjusts the candidate matching point pairs based on the kernel density to obtain the target matching point pairs, thereby balancing the distribution of matching points to retain more original scene information, thereby effectively avoiding the deviation caused by excessive repeated sampling of high-density areas in the preliminary sampling, and finally generating a set of more comprehensive and evenly distributed balanced samples; and performs 3D reconstruction of the target parking scene based on the target matching point pairs, thereby improving the accuracy and robustness of the scene reconstruction, and avoiding the situation where the reconstruction result is inconsistent with the actual scene.

[0083] Figure 6 FIG. 1 is a flow chart showing another embodiment of the vehicle system software package upgrade method of the present invention, which can be used in the above-mentioned computer device. Figure 6 As shown, the method comprises the following steps:

[0084] Step 610: Acquire multiple key frame image pairs corresponding to the target parking scene.

[0085] For details, please see Figure 1 Step 110 of the illustrated embodiment will not be described in detail here.

[0086] Step 620: extract multiple dense matching point pairs of each key frame image pair, and obtain the confidence and point pair position of each dense matching point pair.

[0087] For details, please see Figure 1 Step 120 of the illustrated embodiment will not be described in detail here.

[0088] Step 630 , obtaining matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions, to obtain a plurality of candidate matching point pairs.

[0089] For details, please see Figure 1 Step 130 of the illustrated embodiment will not be described in detail here.

[0090] Step 640 , calculating the kernel density of multiple candidate matching point pairs, and adjusting the candidate matching point pairs based on the kernel density to obtain target matching point pairs.

[0091] For details, please see Figure 1Step 140 of the illustrated embodiment will not be described in detail here.

[0092] Step 650 , based on the target matching point pairs, three-dimensionally reconstruct the target parking scene to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0093] Specifically, the above step 650 includes:

[0094] Step 6501, obtain the grid model of the target parking scene.

[0095] As described above, by acquiring the grid model of the target parking scene, three-dimensional reconstruction is performed based on the grid model.

[0096] Step 6502, obtaining color data and semantic data of corresponding mesh vertices in the mesh model.

[0097] As described above, by acquiring the color data and semantic data of the corresponding mesh vertices in the mesh model, the color data and semantic data are further analyzed to determine the environmental attributes of each mesh vertex.

[0098] Step 6503, determining the three-dimensional coordinate data of the mesh vertices based on the target matching point pairs.

[0099] As mentioned above, by determining the three-dimensional coordinate data of the grid vertices based on the target matching point pairs, the three-dimensional spatial structure of the parking lot can be accurately mapped. The three-dimensional coordinate data reflects the positional relationship of each object in the parking lot, providing a basis for subsequent scene analysis and processing. In this way, different elements such as vehicles, pedestrians, pillars, walls, etc. in the parking lot can be effectively identified, and their spatial positions and sizes can be distinguished. In addition, the three-dimensional reconstruction results can also be used to simulate and analyze the use of parking lots, such as predicting traffic flow during peak hours, optimizing parking guidance strategies, and evaluating the capacity and layout design of parking lots. By simulating vehicle flow in different scenarios, potential congestion points can be discovered in advance, and the distribution of parking spaces can be adjusted accordingly to improve the efficiency of parking lot use. In addition, three-dimensional reconstruction technology can also assist in security monitoring. By real-time monitoring of vehicle and personnel activities in the parking lot, abnormal situations can be discovered in time to enhance the safety management of the parking lot.

[0100] Step 6504, rendering the grid model based on the three-dimensional coordinate data, color data and semantic data to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0101] Specifically, a semantic map of mesh vertices can be generated based on semantic data; a height map of mesh vertices can be generated based on three-dimensional coordinate data; and the semantic map and the height map can be rendered based on color data to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0102] In actual operation, the sparse reconstruction method is used to obtain the target matching point pairs (preliminary point cloud data) of the target parking scene. The sparse preliminary point cloud data is used to initialize the mesh model, which provides the initial geometric structure and height information for subsequent processing. During the initialization process, each mesh vertex carries multimodal information, including its three-dimensional coordinate data (x, y, z), color data (RGB) and semantic data, to effectively enhance the expressiveness of the mesh, not only reflecting the geometric structure, but also providing rich visual and semantic context.

[0103] In the subsequent stages, this multimodal information is used as input to restore the mesh into an RGB image with semantic annotations and height information through rendering. Specifically, the mesh rendering process generates a corresponding semantic map based on the semantic category information of the vertices, a height map based on the height information, and a realistic texture image combined with the RGB values. These rendering results can intuitively reflect the geometric shape, semantic category, and visual features of the road surface or other targets in the scene.

[0104] Finally, the road mesh generated by this method is not only highly consistent and accurate, but also can be connected to the seamless BEV (bird's eye view) 3D annotation system. This annotation method can significantly simplify the automated annotation workflow, especially in scenarios where large-scale generation of 3D semantic annotation data is required, showing extremely high practicality.

[0105] In addition, by introducing sparse point clouds into mesh generation and semantic annotation, it helps to solve the problems of insufficient height information and difficulty in annotation in related technologies. This not only improves the efficiency and quality of annotation, but also significantly reduces labor costs, providing an efficient solution for 3D reconstruction and automated annotation.

[0106] This embodiment provides a parking lot 3D reconstruction device, such as Figure 7 As shown, including:

[0107] An image pair acquisition module 710 is used to acquire a plurality of key frame image pairs corresponding to a target parking scene;

[0108] The point pair information acquisition module 720 is used to extract multiple dense matching point pairs of each key frame image pair, and obtain the confidence and point pair position of each dense matching point pair;

[0109] A matching point pair acquisition module 730 is used to acquire matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions, among the dense matching point pairs, to obtain a plurality of candidate matching point pairs;

[0110] A kernel density calculation module 740 is used to calculate the kernel density of multiple candidate matching point pairs, and adjust the candidate matching point pairs based on the kernel density to obtain target matching point pairs;

[0111] The parking lot reconstruction module 750 is used to perform three-dimensional reconstruction of the target parking scene based on the target matching point pairs to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0112] In some optional implementations, the image pair acquisition module 710 includes:

[0113] A parking video acquisition unit, used to acquire a parking video shot during the vehicle parking in a target parking scene, wherein the parking video is shot based on a plurality of panoramic cameras arranged on the vehicle;

[0114] A video frame extraction processing unit, used for performing frame extraction processing on the parking video to obtain a parking image set;

[0115] The image pair determination unit is used to determine a plurality of key frame image pairs corresponding to a target parking scene based on the image overlap between adjacent frame parking images in the parking image set.

[0116] In some optional implementations, the point pair information acquisition module 720 includes:

[0117] A feature extraction unit is used to input each key frame image pair into a target encoder for feature extraction to obtain a plurality of dense matching point pairs and corresponding confidence levels of each key frame image pair;

[0118] A semantic segmentation unit, used for performing semantic segmentation on the key frame image pair to obtain the parking lot area where the key frame image pair is located;

[0119] The point pair position determination unit is used to determine the point pair positions of the densely matched point pairs based on the parking lot area where the key frame image pair is located.

[0120] In some optional implementations, the matching point pair acquisition module 730 includes:

[0121] A confidence comparison unit, used to compare the confidence of each dense matching point pair with the target confidence to obtain a confidence comparison result;

[0122] A point pair cleaning unit is used to remove matching point pairs whose confidence is less than the target confidence from the dense matching point pairs based on the confidence comparison result, and obtain cleaning matching point pairs whose confidence is greater than or equal to the target confidence;

[0123] The point pair updating unit is used to obtain the point pair position of the cleaning matching point pair, and use the cleaning matching point pair whose point pair position is the target position as the candidate matching point pair, wherein the target position is used to represent the position for parking in the parking lot area.

[0124] In some optional implementations, the kernel density calculation module 740 includes:

[0125] A statistical analysis unit, used to perform statistical analysis on multiple candidate matching point pairs to obtain kernel density functions corresponding to the multiple candidate matching point pairs;

[0126] A kernel density calculation unit, used to calculate the kernel density of each candidate matching point pair based on the kernel density function;

[0127] A weight calculation unit, used to use the inverse of the kernel density as the weighted weight of the candidate matching point;

[0128] The point pair adjustment unit is used to adjust the candidate matching point pair based on the product of the weighted weight and the candidate matching point pair to obtain the target matching point pair.

[0129] In some optional embodiments, the parking lot reconstruction module 750 includes:

[0130] A posture information acquisition unit, used to acquire the camera posture information corresponding to the key frame image;

[0131] A point cloud data acquisition unit, used to obtain sparse three-dimensional point cloud data of a target parking scene based on camera posture information and target matching point pairs;

[0132] The first three-dimensional reconstruction unit is used to perform three-dimensional reconstruction on the target parking scene based on the sparse three-dimensional point cloud data to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0133] In some optional implementations, the parking lot reconstruction module 750 further includes:

[0134] A grid model acquisition unit, used to acquire a grid model of a target parking scene;

[0135] A vertex data acquisition unit, used to acquire color data and semantic data corresponding to mesh vertices in the mesh model;

[0136] A coordinate data determination unit, used to determine three-dimensional coordinate data of mesh vertices based on target matching point pairs;

[0137] The second three-dimensional reconstruction unit is used to render the grid model based on the three-dimensional coordinate data, the color data and the semantic data to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

[0138] The parking lot 3D reconstruction device provided by the present embodiment obtains a target image block and a corresponding search area image block in a video; performs feature extraction on the target image block to obtain image features, and performs feature extraction on the search area image block to obtain regional image features; after cascading the image features and the regional image features, feature extraction is performed through an attention network to obtain target image features, so as to realize information interaction between the graphic features and the regional image features, thereby improving the efficiency and reliability of feature extraction; and obtains a target image frame based on the target image features to track the target, thereby improving the accuracy and stability of target tracking.

[0139] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0140] The parking lot 3D reconstruction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0141] The embodiment of the present invention also provides a computer device having the above Figure 7 The parking lot 3D reconstruction device is shown.

[0142] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 810, memory 820, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 810 is taken as an example.

[0143] The processor 810 may be a central processing unit, a network processor or a combination thereof. The processor 810 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0144] The memory 820 stores instructions executable by at least one processor 810, so that the at least one processor 810 executes the method shown in the above embodiment.

[0145] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 820 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 820 may optionally include a memory remotely arranged relative to the processor 810, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The memory 820 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 820 may also include a combination of the above types of memory.

[0147] The computer device also includes a communication interface 830 for the computer device to communicate with other devices or a communication network.

[0148] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0149] An embodiment of the present invention further provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute the parking lot three-dimensional reconstruction method of the first aspect or any corresponding embodiment thereof.

[0150] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A parking lot 3D reconstruction method, characterized in that: The method comprises: Acquire multiple key frame image pairs corresponding to the target parking scene; Extracting a plurality of dense matching point pairs of each of the key frame image pairs, and obtaining the confidence and point pair position of each of the dense matching point pairs; Acquire matching point pairs whose confidences are greater than or equal to target confidences and whose point pair positions are target positions, in the dense matching point pairs, to obtain a plurality of candidate matching point pairs; Calculating the kernel density of the plurality of candidate matching point pairs, and adjusting the candidate matching point pairs based on the kernel density to obtain target matching point pairs; Based on the target matching point pairs, the target parking scene is three-dimensionally reconstructed to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of key frame image pairs corresponding to the target parking scene includes: Acquire a parking video captured by the vehicle during parking in the target parking scene, wherein the parking video is captured by multiple panoramic cameras provided on the vehicle; Performing frame extraction processing on the parking video to obtain a parking image set; Based on the image overlap between adjacent parking image frames in the parking image set, a plurality of key frame image pairs corresponding to the target parking scene are determined.

3. The method according to claim 1, characterized in that The step of extracting a plurality of dense matching point pairs of each of the key frame image pairs and obtaining the confidence level and point pair position of each of the dense matching point pairs includes: Inputting each of the key frame image pairs into a target encoder for feature extraction to obtain a plurality of dense matching point pairs and corresponding confidence levels of each of the key frame image pairs; Performing semantic segmentation on the key frame image pair to obtain the parking lot area where the key frame image pair is located; Based on the parking lot area where the key frame image pair is located, the point pair positions of the dense matching point pairs are determined.

4. The method according to claim 1, characterized in that: The obtaining of the matching point pairs in which the confidence is greater than or equal to the target confidence and the point pair positions are the target positions, so as to obtain a plurality of candidate matching point pairs, includes: Comparing the confidence of each of the dense matching point pairs with the target confidence to obtain a confidence comparison result; Based on the confidence comparison result, the matching point pairs whose confidence is less than the target confidence are eliminated from the dense matching point pairs, so as to obtain the cleaned matching point pairs whose confidence is greater than or equal to the target confidence; The point pair position of the cleaning matching point pair is obtained, and the cleaning matching point pair whose point pair position is a target position is used as the candidate matching point pair, where the target position is used to represent a position for parking in a parking lot area.

5. The method according to claim 1, characterized in that The calculating the kernel density of the plurality of candidate matching point pairs and adjusting the candidate matching point pairs based on the kernel density to obtain target matching point pairs includes: Performing statistical analysis on the multiple candidate matching point pairs to obtain kernel density functions corresponding to the multiple candidate matching point pairs; Calculating the kernel density of each of the candidate matching point pairs based on the kernel density function; Using the inverse of the kernel density as the weighted weight of the candidate matching point; The candidate matching point pair is adjusted based on the product of the weighted weight and the candidate matching point pair to obtain the target matching point pair.

6. The method according to claim 1, characterized in that The three-dimensional reconstruction of the target parking scene based on the target matching point pair to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene includes: Obtaining camera posture information corresponding to the key frame image; Based on the camera posture information and the target matching point pairs, sparse three-dimensional point cloud data of the target parking scene is obtained; The target parking scene is three-dimensionally reconstructed based on the sparse three-dimensional point cloud data to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

7. The method according to claim 1, characterized in that The three-dimensional reconstruction of the target parking scene based on the target matching point pair to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene also includes: Acquire a grid model of the target parking scene; Obtaining color data and semantic data of corresponding mesh vertices in the mesh model; Determine the three-dimensional coordinate data of the mesh vertices based on the target matching point pairs; The grid model is rendered based on the three-dimensional coordinate data, the color data and the semantic data to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

8. A parking lot 3D reconstruction device, characterized in that: The device comprises: An image pair acquisition module, used to acquire multiple key frame image pairs corresponding to a target parking scene; A point pair information acquisition module, used to extract multiple dense matching point pairs of each key frame image pair, and obtain the confidence and point pair position of each dense matching point pair; A matching point pair acquisition module, used to acquire matching point pairs whose confidence is greater than or equal to the target confidence and whose point pair positions are the target positions, in the dense matching point pairs, to obtain a plurality of candidate matching point pairs; A kernel density calculation module, used to calculate the kernel density of the plurality of candidate matching point pairs, and adjust the candidate matching point pairs based on the kernel density to obtain target matching point pairs; The parking lot reconstruction module is used to perform three-dimensional reconstruction of the target parking scene based on the target matching point pair to obtain a three-dimensional reconstruction result of the parking lot of the target parking scene.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the parking lot three-dimensional reconstruction method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the parking lot three-dimensional reconstruction method according to any one of claims 1 to 8.

Images