Automatic memory parking front-end matching method and system, electronic equipment and medium

Abstract

The invention provides an automatic memory parking front-end matching method and system, electronic equipment and a medium, and belongs to the technical field of automatic driving, and the method comprises the steps: obtaining observation data based on a monocular camera, and constructing a motion and observation model according to the observation data; constructing visual point cloud data according to the observation data; constructing a probability grid map, and constructing a sub-map based on the probability grid map; and matching the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle. Automatic memory parking front end matching is carried out only by using pure visual information of the monocular camera, self-positioning of the vehicle is completed, the method does not depend on a high-definition map and a laser radar sensor, and the cost of a traditional automatic driving scheme is greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an automatic memory parking front-end matching method, system, electronic device, and medium. Background Technology

[0002] With the development of autonomous driving technology, automatic parking has become an important component of modern intelligent vehicles. However, traditional automatic parking solutions often rely on preset high-definition maps or LiDAR sensors, which have high hardware costs and require significant computing power from the domain controller.

[0003] In vehicle autonomous driving technology, traditional solutions achieve environmental perception and localization through laser point cloud matching. However, laser point cloud matching technology also has limitations and challenges such as high computational complexity and high hardware cost. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes an automatic memory parking front-end matching method, system, electronic device and medium.

[0005] In a first aspect, embodiments of the present invention provide an automatic memory parking front-end matching method, comprising:

[0006] A motion and observation model is constructed based on observation data acquired using a monocular camera.

[0007] Visual point cloud data is constructed based on the observed data;

[0008] Construct a probabilistic raster map, and then construct a sub-map based on the probabilistic raster map;

[0009] Based on the motion and observation model, the visual point cloud data is matched with the sub-map to obtain the vehicle's current pose.

[0010] In some embodiments, the step of acquiring observation data based on a monocular camera and constructing a motion and observation model based on the observation data includes:

[0011] Observational data for the current and previous moments are acquired using a monocular camera;

[0012] The vehicle position change information between the previous moment and the current moment is obtained based on inertial sensors and odometers;

[0013] A motion and observation model is constructed based on the observation data and vehicle position change information.

[0014] In some embodiments, acquiring observation data from the current and previous moments based on a monocular camera includes:

[0015] It acquires current observation data of the surrounding environment based on a monocular camera;

[0016] Historical observation data of the previous moment at the prior position is obtained using a monocular camera;

[0017] Accordingly, a motion and observation model is constructed based on the observation data and vehicle position change information, including:

[0018] A motion and observation model is established based on the prior location, historical observation data, vehicle position change information, and current observation data.

[0019] In some embodiments, constructing visual point cloud data based on the observation data includes:

[0020] Two-dimensional image information is obtained based on the observation data;

[0021] Create a 2D pseudo-point cloud based on the two-dimensional image information;

[0022] The 2D pseudo-point cloud is projected onto the direction of gravity to obtain visual point cloud data.

[0023] In some embodiments, constructing a probabilistic raster map and constructing a sub-map based on the probabilistic raster map includes:

[0024] Set the initial raster map;

[0025] The occupancy probability of the initial grid map is updated in real time based on the image observation results to obtain a probabilistic grid map;

[0026] A sub-map is constructed based on the probabilistic raster map.

[0027] In some embodiments, matching the visual point cloud data with the sub-map based on the motion and observation model to obtain the vehicle's current pose includes:

[0028] Based on the motion and observation model, the matching of the visual point cloud data and the sub-map is transformed into a maximum a posteriori probability estimation problem;

[0029] The maximum posterior probability estimation problem is optimized and solved to obtain the maximum probability pose of the vehicle in the current state;

[0030] The vehicle's maximum probability pose in the current state is taken as the vehicle's current pose.

[0031] In some embodiments, optimizing the maximum posterior probability estimation problem to obtain the maximum probability pose of the vehicle in the current state includes:

[0032] Residual blocks are defined based on the differences between the visual point cloud data and the sub-map;

[0033] Based on the residual block, the maximum a posteriori probability estimation problem is transformed into a nonlinear least squares problem and optimization problem with boundary constraints;

[0034] The nonlinear least squares problem with boundary constraints and the optimization problem are solved using a preset optimization library to obtain the maximum probability pose of the vehicle in the current state.

[0035] Secondly, embodiments of the present invention provide an automatic memory parking front-end matching system, comprising:

[0036] The model building module is used to acquire observation data based on a monocular camera and build a motion and observation model based on the observation data.

[0037] The point cloud construction module is used to construct visual point cloud data based on the observation data;

[0038] The map building module is used to construct a probabilistic raster map and to construct sub-maps based on the probabilistic raster map.

[0039] The matching and solving module is used to match the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle.

[0040] Thirdly, embodiments of the present invention provide an electronic device, including:

[0041] One or more processors;

[0042] Memory, used to store one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described above.

[0044] Fourthly, embodiments of the present invention provide a computer-readable medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0045] The automatic memory parking front-end matching method provided by this invention includes: acquiring observation data based on a monocular camera; constructing a motion and observation model based on the observation data; constructing visual point cloud data based on the observation data; constructing a probabilistic grid map; constructing a sub-map based on the probabilistic grid map; and matching the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle. This invention achieves vehicle self-localization by using only pure visual information from a monocular camera for automatic memory parking front-end matching, without relying on high-definition maps and LiDAR sensors, significantly reducing the cost of traditional autonomous driving solutions. Attached Figure Description

[0046] Figure 1 A flowchart illustrating an automatic memory parking front-end matching method provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the motion and observation model involved in the embodiments of the present invention;

[0048] Figure 3 This is a schematic diagram of a 2D point cloud involved in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the probability grid map involved in the embodiments of the present invention;

[0050] Figure 5 This is a schematic diagram of the first code involved in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the second code involved in an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the third code involved in the embodiments of the present invention;

[0053] Figure 8 This is a schematic diagram of the fourth code involved in the embodiments of the present invention;

[0054] Figure 9 This is a schematic diagram of the fifth code involved in the embodiments of the present invention;

[0055] Figure 10 This is a schematic diagram of the sixth code involved in the embodiments of the present invention;

[0056] Figure 11 This is a schematic diagram of the seventh code involved in the embodiments of the present invention;

[0057] Figure 12 This is a structural block diagram of an automatic memory parking front-end matching system provided in an embodiment of the present invention;

[0058] Figure 13 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0060] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0061] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0062] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0063] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0064] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0065] The key terms involved in this invention are defined as follows:

[0066] AVP: Automated Valet Parking; Memory Parking is an advanced driver assistance technology designed to enable vehicles to park automatically without a driver. This technology utilizes various sensors, cameras, radar, and high-precision maps, combined with advanced algorithms, to achieve autonomous navigation and parking of the vehicle.

[0067] SLAM: Simultaneous Localization and Mapping. SLAM technology is a key technology in fields such as robotics and autonomous vehicles. It enables devices to simultaneously build a map of the environment and determine their own location while moving in unknown environments, allowing them to navigate and explore autonomously. SLAM implementation typically consists of two main parts: a front-end and a back-end.

[0068] Front-end: In the state estimation problem of SLAM, the front-end is responsible for processing perceived data to identify environmental features and track changes in the position of vehicles or equipment. A crucial task in the front-end is to extract features from image data for matching, and then calculate the sensor's position and orientation relative to the environment based on the matched features, i.e., updating the vehicle's attitude.

[0069] Laser point cloud: A laser point cloud is a set of spatial points representing the characteristics of the surrounding environment. It is generated by a lidar device emitting a laser beam and receiving signals reflected from the surface of an object. Based on information such as time difference and phase difference, parameters such as the target's distance and angle are calculated. Each point represents the positional information of a measured object in space.

[0070] Visual point cloud: Visual point cloud refers to the process of acquiring environmental information through cameras or other visual sensors and converting it into a three-dimensional point cloud data structure. This technology combines computer vision and 3D geometry processing, enabling machines to "see" and understand information such as the shape, size, and location of their surroundings. Visual point clouds have wide applications in many fields, including autonomous driving, robot navigation, augmented reality, and virtual reality.

[0071] Point cloud matching: Point cloud matching refers to finding correspondences between two or more sets of point cloud data acquired at different times or locations to determine their relative transformations (such as rotation and translation). This is crucial for autonomous vehicles because it helps them understand their changes relative to their previous positions, thereby achieving precise positioning.

[0072] Pose: This is a combination of position and orientation. Position refers to the vehicle's specific coordinates in space, typically represented by a three-dimensional Cartesian coordinate system (x, y, z). Orientation describes how the vehicle rotates or tilts relative to its position, i.e., the orientation of the object. In autonomous driving technology, understanding the vehicle's current pose is essential for accurate localization and map matching, route planning, and dynamic obstacle avoidance.

[0073] In relevant vehicle autonomous driving technologies, traditional solutions achieve environmental perception and localization through laser point cloud matching. This primarily involves generating 3D point cloud data of the surrounding environment using LiDAR sensors, and then utilizing this data for accurate environmental modeling (map building) and vehicle localization. However, while laser point cloud matching technology plays a crucial role in autonomous driving, it also faces some limitations and challenges:

[0074] 1. High computational complexity: Especially when processing large-scale point cloud data, the computational load of the algorithm can be very large. For example, the ICP algorithm needs to iteratively find the nearest point pairs and calculate the transformation matrix, which may become a bottleneck in real-time applications.

[0075] 2. High hardware costs: High-quality LiDAR equipment is expensive, and due to the large-scale point cloud data measurement, a high-computing-power domain controller is required, which increases the overall system cost and limits its widespread application.

[0076] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides an automatic memory parking front-end matching method. Figure 1 This is a flowchart illustrating an automatic memory parking front-end matching method provided in an embodiment of the present invention.

[0077] As one embodiment of the present invention, such as Figure 1 As shown, the automatic memory parking front-end matching method includes:

[0078] Step S1: Acquire observation data based on a monocular camera, and construct a motion and observation model based on the observation data;

[0079] Step S2: Construct visual point cloud data based on the observed data;

[0080] Step S3: Construct a probabilistic raster map, and construct a sub-map based on the probabilistic raster map;

[0081] Step S4: Based on the motion and observation model, match the visual point cloud data with the sub-map to obtain the current pose of the vehicle.

[0082] It should be noted that the execution subject in this embodiment can be an electronic device, which can be a computer device with data processing function, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, the execution subject is a computer device as an example for explanation.

[0083] This embodiment proposes an automatic memory parking front-end matching method, which is a front-end matching algorithm based on visual point clouds. By transforming the matching of point cloud data with a probabilistic grid map into a maximum a posteriori probability estimation problem, it can be optimized and solved using Ceres to obtain the maximum probability pose of the vehicle in the current state. In an autonomous parking system, obtaining the vehicle's current pose is crucial for mapping and localization; therefore, the matching algorithm in this embodiment is one of the core algorithms of autonomous parking technology.

[0084] Specifically, the technical solution process in this embodiment includes: establishing a motion and observation model → constructing a visual point cloud → designing a probabilistic raster map → designing a sub-map → matching. The matching algorithm in this embodiment first establishes and maintains a series of sub-maps frame by frame, where each sub-map is essentially a probabilistic raster map. When a new image frame is received, the matching algorithm estimates the vehicle's maximum possible pose (position + attitude) in the sub-map based on the current image information, and then transmits this pose to the downstream module for mapping and localization. The specific steps are described below.

[0085] In some embodiments, acquiring observation data based on a monocular camera and constructing a motion and observation model based on the observation data includes: acquiring observation data at the current moment and the previous moment based on the monocular camera; obtaining vehicle position change information between the previous moment and the current moment based on an inertial sensor and an odometer; and constructing a motion and observation model based on the observation data and the vehicle position change information.

[0086] In some embodiments, acquiring observation data at the current moment and the previous moment based on a monocular camera includes: acquiring current observation data of the surrounding environment at the current moment based on the monocular camera; acquiring historical observation data of the previous moment at a priori position based on the monocular camera; and correspondingly, constructing a motion and observation model based on the observation data and vehicle position change information includes: establishing a motion and observation model based on the priori position, historical observation data, vehicle position change information, and current observation data.

[0087] Specifically, such as Figure 2 As shown, a motion and observation model is established: when a vehicle moves using a visual sensor, it is assumed that at time i... The location is observed through the camera, creating a view of the surrounding environment. In the previous moment The observations generated by the location are The change in vehicle position between two adjacent time points (time i and time i-1) can be sensed by an inertial measurement unit (IMU) and an odometer. The pose change (vehicle position change information) can be denoted as u. Since the pose and observation of the previous time point are known, and the observation at time i is also known, this mathematical model (motion and observation model) can be expressed as: Given... , u Given this premise, how can we derive the result as accurately as possible? This means finding the pose at time i. Essentially, this is a maximum a posteriori probability estimation problem, i.e., finding p( , u The maximum value of ).

[0088] For example, at any time i, the state of the vehicle is represented by... Representation includes position (e.g., two-dimensional or three-dimensional coordinates) and orientation (e.g., direction or rotation). Observation Image or point cloud data of the surrounding environment captured at time i by a visual sensor (e.g., a monocular camera). At the previous time i-1, the vehicle's position and observations are represented as follows: and Vehicle observation at time i Based on its current location And Based on It is predicted based on the pose change u. The state at the previous moment is known. and observation And the observation at the current moment Using this known information to estimate the state at the current moment. This can be achieved by constructing a probabilistic model, including a state transition model (describing how the state changes over time) and an observation model (describing how observations are generated from the state). Using Bayes' theorem, the problem can be expressed as solving for the posterior probability p(…). , u The maximum value of ).

[0089] In some embodiments, constructing visual point cloud data based on the observation data includes: obtaining two-dimensional image information based on the observation data; creating a 2D pseudo-point cloud based on the two-dimensional image information; and projecting the 2D pseudo-point cloud onto the gravity direction to obtain visual point cloud data.

[0090] Specifically, visual point cloud construction: such as Figure 3As shown, visual point clouds based on camera data refer to the process of generating three-dimensional point clouds by processing two-dimensional image information and utilizing computer vision technology. The method used in this embodiment does not rely on depth sensors such as LiDAR or stereo cameras; it constructs point clouds using only images acquired by a monocular camera. In this embodiment, the monocular camera acquires 2D pseudo-point clouds, which, after being projected onto the gravity direction, have their Z-axis values ​​uniformly set to 0. This point cloud data (visual point cloud data) is then passed as parameters to the subsequent matching function for matching.

[0091] For example, a monocular camera captures two-dimensional images, where each pixel represents a two-dimensional projection of an object from the camera's perspective. To recover three-dimensional information from these two-dimensional images, additional information is typically needed, such as the camera's intrinsic parameters (focal length, principal point, etc.) and extrinsic parameters (camera position and orientation). Without depth information, 2D pseudo-point clouds can be created. The X and Y coordinates of these point clouds are extracted from the images, while the Z coordinate is typically set to 0, indicating that the points lie on the same plane. 2D pseudo-point clouds can be projected onto the direction of gravity, meaning that all points in the cloud are assumed to be aligned in the direction of gravity; their height information is ignored or uniformly set to 0. This projection simplifies the problem, allowing the point cloud to include only horizontal position information.

[0092] For example, the constructed 2D pseudo-point cloud data (visual point cloud data) can then be passed to a matching function, which matches the point cloud against a known environment map (e.g., a raster map) to estimate the vehicle's position and orientation. The matching function employs optimization techniques (e.g., Ceres Solver) to minimize the discrepancies between the point cloud and the map, thereby estimating the location of the best match.

[0093] In some embodiments, constructing a probabilistic grid map and constructing a sub-map based on the probabilistic grid map includes: setting an initial grid map; updating the occupancy probability of the initial grid map in real time according to image observation results to obtain the probabilistic grid map; and constructing a sub-map based on the probabilistic grid map.

[0094] Specifically, probabilistic grid map design: such as Figure 4 As shown, a grid map is an abstract form of map that divides the world into grids, each called a grid cell. A single grid cell has only two states: occupied or empty. Machine observation of the world is full of errors; directly assuming a grid cell is either occupied or empty is inaccurate and needs to be represented by probability. For a given grid cell on the map... , using p ( ) represents the probability that it is occupied (called the occupancy probability), denoted by p( The occupancy probability () represents the probability of the grid being idle. For a grid that has not yet been observed, its initial occupancy probability and idle probability are both set to 0.5 by default. The ratio of the occupancy probability to the idle probability is used to characterize the grid's state, denoted as odds, i.e.:

[0095] odds( )=p( ) / p(- ) = p( ) / (1-p( ))

[0096] Considering that addition is much faster than multiplication in a computer CPU, and that logarithmic operations can convert multiplication into addition, we take the logarithm of odds, denoted as: l( =log(odds( This uses l( ) represents a grid The probability of being occupied, expressed as a logarithmic probability, is called a probability grid map.

[0097] For example, after setting the initial grid map, the occupancy probability of the grid map needs to be updated in real time based on the image observation results from the camera. For time t-1, the occupancy probability of the grid is known to be... ( ), the probability of observing z at time t is P( ).

[0098] The probability update formula for a raster map is:

[0099]

[0100] in, ( (The initial time) The probability of occupancy is easy to know. Therefore, the probability update formula for a raster map is:

[0101]

[0102] In this way, each probability update only requires an addition calculation based on the probability of the previous time step.

[0103] Specifically, the submap design uses consecutive image frames captured by the camera to create submaps. These submaps employ a probabilistic raster format, facilitating easy matching. Each submap class maintains two submap objects: an active submap for inserting new image frame data and an old submap for matching. Once the active submap has undergone initialization with a certain amount of data inserted, it is treated as the old submap for matching. The original old submap is discarded, and a new active submap is reconstructed.

[0104] Understandably, a submap can be a local view of a global probabilistic raster map, including a small portion of the environment surrounding the vehicle. The design of the submap allows for a focus on the immediate environment, thus improving the efficiency and accuracy of matching and localization. The submap takes the form of a probabilistic raster and updates its state in real time based on new observation data, reflecting changes in the environment. The submap class maintains two submap objects: an active submap and an old submap. The active submap is used to insert new image frame data in real time, which is then used to update the submap's state. The old submap is used for matching operations, i.e., comparing with new image data to estimate the vehicle's pose.

[0105] For example, once a certain amount of data has been inserted into the active subgraph, the initialization operation is considered complete. Once initialization is complete, the active subgraph becomes the old subgraph, used for matching with new image data. Simultaneously, the original old subgraph is discarded, and a new active subgraph is reconstructed to prepare for receiving the next batch of new image frames. This process is a cyclical update process; the active subgraph continuously receives new data, updates itself, becomes the old subgraph for matching, and a new active subgraph is created. This design allows the vehicle to continuously update its local map and perform matching operations with each new image frame, thereby enabling the vehicle to continuously perceive and locate its surroundings.

[0106] In some embodiments, matching the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle includes: transforming the matching of the visual point cloud data with the sub-map into a maximum a posteriori probability estimation problem based on the motion and observation model; optimizing and solving the maximum a posteriori probability estimation problem to obtain the maximum probability pose of the vehicle in the current state; and using the maximum probability pose of the vehicle in the current state as the current pose of the vehicle.

[0107] In some embodiments, optimizing the maximum a posteriori probability estimation problem to obtain the maximum probability pose of the vehicle in the current state includes: defining residual blocks based on the difference between the visual point cloud data and the sub-map; transforming the maximum a posteriori probability estimation problem into a nonlinear least squares problem and an optimization problem with boundary constraints based on the residual blocks; and solving the nonlinear least squares problem and the optimization problem with boundary constraints according to a preset optimization library to obtain the maximum probability pose of the vehicle in the current state.

[0108] In this embodiment, the focus and main innovation lie in the design of the matching algorithm and the construction of the C++ program. The matching algorithm can use the Ceres optimization library to solve nonlinear least squares problems and optimization problems with boundary constraints.

[0109] Specifically, the general mathematical expression for Ceres optimization computation is as follows:

[0110]

[0111] In the equation, Represents a residual block, where, () represents the cost function, which is the primary objective of the optimization calculation; This is a kernel function used to filter out the influence of extreme value errors on the results; it optimizes parameters. This is the parameter block, i.e., the required solution.

[0112] It should be noted that the goal of front-end matching is to accurately estimate the vehicle's current pose. First, estimate the observation model regarding The probability distribution, the pose corresponding to the observation model taking its maximum value is... The most accurate estimate. By describing the problem in the form of the general mathematical expression formula for Ceres optimization calculation described above, it can be solved using Ceres.

[0113] For example, the main steps of solving using Ceres in this embodiment are: a) Define the residual blocks to be minimized, each residual block corresponding to a cost function. b) Create a Problem object and add these residual blocks to it. c) Set solver options, such as selecting an optimization algorithm and setting termination conditions. d) Call the Solve() function to perform optimization.

[0114] In one example, reference Figures 5 to 11 The code shown illustrates the matching algorithm and program design:

[0115] like Figure 5As shown, a match function is first designed as the main entry point of the algorithm program. This function accepts 6 parameters, namely the coordinates of the vehicle's prior position, the optimized starting pose, the point cloud used for matching, the grid map used for matching, the optimized pose output after matching, and some log records during the Ceres calculation process.

[0116] like Figure 6 As shown, three objects are defined in the match function: problem, options, and summary. problem describes the problem to be solved, options provides configuration options, and summary records the solution process.

[0117] like Figure 7 As shown, the key to Ceres computation is designing the cost function calculation method and defining each residual block and kernel function. Residual blocks are added using problem objects.

[0118] like Figure 8 As shown, the AddResidualBlock interface has three parameters: the cost function, the kernel function, and the pose to be optimized. In this embodiment's matching algorithm, the kernel function is set to a null pointer. The cost function design must provide a method for calculating the residuals. Calculating the residuals requires constructing a functor class and providing an overloaded version of the parentheses operator "()". Returning true after the residual calculation is complete indicates that the calculation is finished.

[0119] like Figure 9 As shown, designing the cost function also requires specifying its differentiation method in Ceres and providing the specific implementation of the function. This embodiment uses automatic differentiation in Ceres, which converges faster than numerical differentiation.

[0120] like Figure 10 As shown, the matching algorithm in this embodiment has three types of residuals: the occupancy probability of the grid, translation, and rotation. After all three residuals are added to the problem object, ceres::Solve can be called to solve it.

[0121] like Figure 11 As shown, after the solution is completed, the optimized pose is recorded in the pointer and used as the return value of the match function.

[0122] At this point, the entire scan matching calculation process is complete, and the vehicle has obtained the optimized latest pose (the vehicle's current pose).

[0123] It should be noted that the matching algorithm and program design are as follows: A function named `match` is defined as the main entry point of the algorithm program. This function accepts the following six parameters: the coordinates of the vehicle's prior position (a two-dimensional vector), the optimized initial pose (a rigid body transformation object), the point cloud data used for matching, the raster map used for matching, a pointer to the optimized pose output after matching (a pointer to a rigid body transformation object), and a pointer to the log records during the Ceres computation process (a pointer to the Ceres solver summary). Inside the `match` function, the following Ceres-related objects are initialized: a `ceres::Problem` object is created to describe and store all information about the optimization problem; a `ceres::Solver::Options` object is created to set the solver's configuration options; and a `ceres::Solver::Summary` object is created to record relevant information about the solution process. Three different residual blocks are added using the Ceres problem object, corresponding to: raster occupancy probability residuals, translation residuals, and rotation residuals. For each residual, the problem's AddResidualBlock method is called, passing in the corresponding cost function, kernel function (here, a null pointer), and optimized pose. A CostFunctor class is designed, which needs to overload the operator "()" to provide a method for calculating the residuals. In the overloaded operator, the point cloud data is traversed, and the residual is calculated for each point. A CreateCostFunction function is implemented, which accepts some parameters (such as scaling factor, point cloud, and raster map), then creates and returns a new ceres::AutoDiffCostFunction object, which uses the CostFunctor class for automatic differentiation. Options for the Ceres solver are set, including selecting the solver type, linear algebra library, and other optimization parameters. The ceres::Solve function is called, passing in the previously set solver options, the problem object, and the summary object to begin the solution process. After the solution is complete, the optimized pose is extracted from the Ceres Solver result and assigned to the optimized pose pointer provided in the match function parameter. Once the match function finishes execution, it returns the optimized pose, which is the latest vehicle pose calculated by the algorithm (the vehicle's current pose). The entire process concludes here; the match function utilizes Ceres Solver to perform optimized estimation of the vehicle pose.

[0124] In this embodiment, matching is based on visual point clouds, without relying on traditional high-definition maps and LiDAR sensors. The vehicle's self-localization is completed using only the pure visual information from a monocular camera, which significantly reduces the cost of traditional autonomous driving solutions.

[0125] The automatic memory parking front-end matching method provided in this embodiment includes: acquiring observation data based on a monocular camera; constructing a motion and observation model based on the observation data; constructing visual point cloud data based on the observation data; constructing a probabilistic grid map; constructing a sub-map based on the probabilistic grid map; and matching the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle. This embodiment achieves vehicle self-localization by using only pure visual information from a monocular camera for automatic memory parking front-end matching, without relying on high-definition maps and LiDAR sensors, significantly reducing the cost of traditional autonomous driving solutions.

[0126] Reference Figure 12 , Figure 12 This is a structural block diagram of an embodiment of the automatic memory parking front-end matching system of the present invention. Figure 12 As shown, the automatic memory parking front-end matching system includes:

[0127] Model building module 10 is used to acquire observation data based on a monocular camera and build a motion and observation model based on the observation data;

[0128] Point cloud construction module 20 is used to construct visual point cloud data based on the observation data;

[0129] The map building module 30 is used to build a probabilistic raster map and to build a sub-map based on the probabilistic raster map.

[0130] The matching and solving module 40 is used to match the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle.

[0131] The automatic memory parking front-end matching system provided in this embodiment automatically remembers parking front-end matching by using only pure visual information from a monocular camera to complete the vehicle's self-localization. It does not rely on high-definition maps and LiDAR sensors, which greatly reduces the cost of traditional autonomous driving solutions.

[0132] In addition, for technical details not described in detail in this embodiment of the automatic memory parking front-end matching system, please refer to the automatic memory parking front-end matching method provided in any embodiment of the present invention, which will not be repeated here.

[0133] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 13 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 13As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the automatic memory parking front-end matching methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0134] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0135] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0136] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0137] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the automatic memory parking front-end matching methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0138] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described automatic memory parking front-end matching method.

[0139] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0140] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0141] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0142] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0143] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0144] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0146] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0148] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. An automatic memory parking front-end matching method, characterized in that, include: A motion and observation model is constructed based on observation data acquired using a monocular camera. Visual point cloud data is constructed based on the observed data; Construct a probabilistic raster map, and then construct a sub-map based on the probabilistic raster map; Based on the motion and observation model, the visual point cloud data is matched with the sub-map to obtain the vehicle's current pose.

2. The method according to claim 1, characterized in that, The process of acquiring observation data based on a monocular camera and constructing a motion and observation model based on the observation data includes: Observational data for the current and previous moments are acquired using a monocular camera; The vehicle position change information between the previous moment and the current moment is obtained based on inertial sensors and odometers; A motion and observation model is constructed based on the observation data and vehicle position change information.

3. The method according to claim 2, characterized in that, The acquisition of observation data at the current and previous moments based on a monocular camera includes: It acquires current observation data of the surrounding environment based on a monocular camera; Historical observation data of the previous moment at the prior position is obtained using a monocular camera; Accordingly, a motion and observation model is constructed based on the observation data and vehicle position change information, including: A motion and observation model is established based on the prior location, historical observation data, vehicle position change information, and current observation data.

4. The method according to claim 1, characterized in that, The construction of visual point cloud data based on the observation data includes: Two-dimensional image information is obtained based on the observation data; Create a 2D pseudo-point cloud based on the two-dimensional image information; The 2D pseudo-point cloud is projected onto the direction of gravity to obtain visual point cloud data.

5. The method according to claim 1, characterized in that, The construction of the probabilistic raster map, and the construction of sub-maps based on the probabilistic raster map, includes: Set the initial raster map; The occupancy probability of the initial grid map is updated in real time based on the image observation results to obtain a probabilistic grid map; A sub-map is constructed based on the probabilistic raster map.

6. The method according to claim 1, characterized in that, The step of matching the visual point cloud data with the sub-map based on the motion and observation model to obtain the vehicle's current pose includes: Based on the motion and observation model, the matching of the visual point cloud data and the sub-map is transformed into a maximum a posteriori probability estimation problem; The maximum posterior probability estimation problem is optimized and solved to obtain the maximum probability pose of the vehicle in the current state; The vehicle's maximum probability pose in the current state is taken as the vehicle's current pose.

7. The method according to claim 6, characterized in that, The optimization solution to the maximum a posteriori probability estimation problem to obtain the maximum probability pose of the vehicle in the current state includes: Residual blocks are defined based on the differences between the visual point cloud data and the sub-map; Based on the residual block, the maximum a posteriori probability estimation problem is transformed into a nonlinear least squares problem and optimization problem with boundary constraints; The nonlinear least squares problem with boundary constraints and the optimization problem are solved using a preset optimization library to obtain the maximum probability pose of the vehicle in the current state.

8. An automatic memory parking front-end matching system, characterized in that, include: The model building module is used to acquire observation data based on a monocular camera and build a motion and observation model based on the observation data. The point cloud construction module is used to construct visual point cloud data based on the observation data; The map building module is used to construct a probabilistic raster map and to construct sub-maps based on the probabilistic raster map. The matching and solving module is used to match the visual point cloud data with the sub-map based on the motion and observation model to obtain the current pose of the vehicle.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images