Spatial positioning method and system based on fusion of monocular vision ranging and laser point cloud
By introducing laser point cloud fusion into monocular visual ranging technology, using two-dimensional projection and three-dimensional environmental model for geometric positioning, the problem of low positioning accuracy of monocular visual ranging in complex environments is solved, and higher positioning is achieved Accuracy and environmental adaptability.
Patent Information
- Application Number
- CN202411124770.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-16
AI Technical Summary
Monocular visual ranging technology has a large impact on the positioning accuracy of light changes, occlusion and parallax in complex or sparse textures. It lacks depth information and global environment information, resulting in inaccurate positioning results and difficult to adapt to complex environments.
Combining monocular visual ranging and laser point cloud technology, a 2D image is received through a monocular sensor and loaded a three-dimensional model for two-dimensional projection, combined with a laser scanner to build a three-dimensional environmental model, and geometric positioning is performed using the acquisition focal length, actual size and image size. , accurately calculate the distance of the target object and locate it in a three-dimensional environment.
It improves the accuracy and reliability of spatial positioning, enhances environmental adaptability, and ensures the precise identification and positioning of target objects in complex environments.
Smart Images

Figure CN118642121B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of spatial positioning technology, and specifically to a spatial positioning method and system that integrates monocular vision ranging with laser point cloud. Background Art
[0002] In modern positioning technology, monocular vision distance measurement is a widely used method that estimates the distance between the target object and the camera by analyzing image information. This method usually relies on image processing and computer vision techniques, such as feature-based methods and deep learning-based methods. Feature-based methods estimate distance by detecting and matching feature points in images, while deep learning-based methods predict depth information by training a large amount of image data.
[0003] Although monocular visual ranging technology is low-cost and easy to implement, it is easily affected by factors such as lighting changes, occlusion and parallax in complex or sparsely textured environments, resulting in limited positioning accuracy. In particular, in the absence of depth information, monocular positioning technology has difficulty in accurately estimating the target object, and due to the lack of global environmental information, its positioning results are often fragmented, making it difficult to effectively analyze the state of the positioning target in the macro environment. In structured environments, such as indoors or outdoor scenes with obvious features, the texture and lighting conditions in the environment may cause the accuracy of feature point matching to decrease, thereby affecting the performance of monocular visual ranging. Summary of the invention
[0004] The present application provides a spatial positioning method and system that integrates monocular vision ranging and laser point cloud, which solves the technical problem that the monocular positioning technology lacks depth information and global environmental information, resulting in limited positioning accuracy and environmental adaptability. By combining monocular vision ranging and laser point cloud technology, the accuracy and reliability of spatial positioning are improved, while better integrating with the actual environment and enhancing environmental adaptability.
[0005] In view of the above problems, on the one hand, the present application provides a spatial positioning method that integrates monocular visual ranging and laser point cloud, the method comprising: receiving a two-dimensional image of a positioning target through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag and an image size tag; loading a three-dimensional model of the positioning target, performing two-dimensional projection according to the first acquisition position tag, and obtaining a two-dimensional projection slice image; performing shape analysis on the two-dimensional projection slice image and the two-dimensional image of the positioning target, and obtaining a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size tag; performing geometric positioning according to the acquisition focal length tag, the actual size tag and the image size tag, and obtaining an image acquisition feature distance; collecting laser point cloud through a laser scanner to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identifier; according to the monocular sensor coordinate identifier, the image acquisition feature distance locates the three-dimensional model of the positioning target in the three-dimensional environment model, and visualizes it through a user end.
[0006] On the other hand, the present application also provides a spatial positioning system that integrates monocular visual ranging and laser point cloud, the system comprising: a monocular ranging module, the monocular ranging module is used to receive a two-dimensional image of a positioning target through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag and an image size tag; a two-dimensional projection module, the two-dimensional projection module is used to load a three-dimensional model of the positioning target, perform two-dimensional projection according to the first acquisition position tag, and obtain a two-dimensional projection slice image; a shape analysis module, the shape analysis module is used to perform shape analysis on the two-dimensional projection slice image and the two-dimensional image of the positioning target, and obtain a selected two-dimensional projection slice image, wherein In the embodiment, the selected two-dimensional projection slice image has an actual size label; a geometric positioning module, the geometric positioning module is used to perform geometric positioning according to the acquisition focal length label, the actual size label and the image size label to obtain the image acquisition feature distance; a three-dimensional environment model construction module, the three-dimensional environment model construction module is used to collect laser point clouds through a laser scanner to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identifier; a target positioning module, the target positioning is used to position the positioning target three-dimensional model in the three-dimensional environment model according to the monocular sensor coordinate identifier and the image acquisition feature distance, and visualize it through the user end.
[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0008] A two-dimensional image of a positioning target is received through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag, and an image size tag. These tags provide the shooting position of the image, the camera focal length, and the size information of the image, and provide basic data for subsequent positioning calculations. The three-dimensional model of the positioning target is loaded, and two-dimensional projection is performed according to the first acquisition position tag to obtain a two-dimensional projection slice image; this step converts the three-dimensional information into a two-dimensional image, and generates a slice image that matches the actual scene, so as to be compared with the image collected by the monocular sensor. The two-dimensional projection slice image and the two-dimensional image of the positioning target are subjected to shape analysis to obtain a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size tag. This step ensures that the selected two-dimensional projection slice image accurately matches the actual image through shape analysis, provides reliable size information, and provides an accurate data basis for geometric positioning. Geometric positioning is performed according to the acquisition focal length tag, the actual size tag, and the image size tag to obtain the image acquisition feature distance. This step accurately calculates the distance information of the target object in the actual space by combining the focal length and image size information, ensures the accuracy of the depth information, and improves the positioning accuracy. Through the laser scanner, the laser point cloud is collected to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identification. This step constructs an accurate three-dimensional environment model, provides rich spatial information and depth data, and enhances the system's environmental perception ability; through the coordinate identification of the monocular sensor, the three-dimensional model of the positioning target can be accurately placed in the three-dimensional environment. According to the coordinate identification of the monocular sensor, the image acquisition feature distance locates the three-dimensional model of the positioning target in the three-dimensional environment model, and visualizes it through the user end. This step merges the three-dimensional model of the positioning target with the three-dimensional environment model, and realizes accurate positioning through coordinate identification and feature distance; and through visual display, the user can intuitively see the positioning result.
[0009] To summarize, the present application combines monocular vision ranging and laser point cloud technology, and provides accurate three-dimensional spatial information through the three-dimensional environment model constructed by the laser scanner. At the same time, the three-dimensional model of the positioning target is used, and compared with the actual captured two-dimensional image through two-dimensional projection, and geometric positioning is performed using the acquisition focal length, actual size and image size. The characteristic distance of image acquisition is calculated more accurately, so that the positioning target can be better integrated with the actual environment, and the accuracy of target recognition and positioning is improved, thereby significantly improving the accuracy and reliability of spatial positioning, and being able to better adapt to complex or changing environments and enhancing environmental adaptability.
[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 A schematic diagram of the flow of a spatial positioning method for integrating monocular vision ranging and laser point cloud provided in an embodiment of the present application;
[0012] Figure 2 A schematic diagram of the process of obtaining a selected two-dimensional projection slice image in the spatial positioning method of the fusion of monocular vision ranging and laser point cloud provided in an embodiment of the present application;
[0013] Figure 3 A schematic diagram of a process for obtaining image acquisition feature distance in a spatial positioning method that combines monocular vision ranging with laser point cloud provided in an embodiment of the present application;
[0014] Figure 4 A schematic diagram of the structure of a spatial positioning system that combines monocular vision ranging with laser point cloud provided in an embodiment of the present application.
[0015] Explanation of the reference numerals: monocular ranging module 10 , two-dimensional projection module 20 , shape analysis module 30 , geometric positioning module 40 , three-dimensional environment model building module 50 , target positioning module 60 . DETAILED DESCRIPTION
[0016] The embodiments of the present application provide a spatial positioning method and system that integrates monocular vision ranging and laser point cloud, thereby solving the technical problem that the monocular positioning technology lacks depth information and global environmental information, resulting in limited positioning accuracy and environmental adaptability. By combining monocular vision ranging and laser point cloud technology, the accuracy and reliability of spatial positioning are improved, while better integrating with the actual environment and enhancing environmental adaptability.
[0017] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides a spatial positioning method integrating monocular visual ranging and laser point cloud, the method comprising:
[0018] Step S1: receiving a two-dimensional image of a positioning target through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag, and an image size tag.
[0019] Specifically, a monocular sensor refers to a camera device with only one lens, such as an ordinary camera or a still camera, which is used to capture two-dimensional images. A two-dimensional image is a planar image captured by a monocular sensor, which only contains the width and height information of the object, but does not contain depth information. The first acquisition position tag is the specific position of the monocular sensor when the image is recorded, such as the coordinate position of the camera in space. The acquisition focal length tag refers to the focal length of the lens used when the image is captured, usually in millimeters (mm), such as a 50mm lens. The image size tag is the resolution or size of the recorded image, such as 1920x1080 pixels.
[0020] First, the monocular sensor is set at a known position and angle and aimed at the target object to shoot. When shooting, the coordinates and focal length of the camera are recorded to form the first acquisition position tag and the acquisition focal length tag. For example, if the camera is fixed on a robot, then the position of the robot and the focal length setting of the camera are the values of these tags. Then, the resolution or size of the image is recorded, that is, the image size tag. Through this step, the monocular sensor records important metadata while acquiring the image. These tags and images provide data basis for subsequent positioning calculations.
[0021] Step S2: Load the three-dimensional model of the positioning target, perform two-dimensional projection according to the first acquisition position label, and obtain a two-dimensional projection slice image.
[0022] Specifically, the three-dimensional model of the positioning target refers to the three-dimensional stereoscopic model of the target object, which contains the structure and shape information of the target object in three-dimensional space. For example, the three-dimensional model of a car includes the length, width, height and shape of the car. Two-dimensional projection is the process of converting a three-dimensional model into a two-dimensional plane image, usually achieved through geometric transformation. This step makes the three-dimensional model appear on a two-dimensional plane. The two-dimensional projection slice image is a two-dimensional image projected from the three-dimensional model, simulating the image taken by the camera from a specific position.
[0023] Use computer-aided design (CAD) software or 3D modeling tools (such as Blender, 3ds Max) to load the 3D model of the target object. The 3D model can be obtained from a pre-built database or created by a 3D scanner. The first acquisition position label, that is, the position of the camera when the image was taken, is used to determine the projection perspective. This perspective is the same as the perspective when the actual image was acquired. For example, if the camera position is in front of the car, the projection angle corresponds to the front view of the car. Then, the 3D model is perspective projected from this perspective, and the geometric transformation algorithm is used to convert the 3D model into a projection image on a 2D plane, that is, a 2D projection slice image. For example, the conversion is performed using the projection function in OpenGL or other graphics libraries. The generated 2D projection slice image is a 2D representation of the 3D model at a specific perspective, which will be used for subsequent shape analysis and positioning. For example, the front view of the car model shows details such as the front face, headlights, and grille of the car.
[0024] Step S3: performing shape analysis on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size label.
[0025] Specifically, shape analysis refers to analyzing the shape of objects in an image to determine the similarity and matching between images. Selecting a two-dimensional projection slice image refers to selecting the one that best matches the actual acquired two-dimensional image from multiple possible projection slice images. This image will be used for subsequent positioning calculations. The actual size label refers to the actual size information of the object in the image, such as the length, width, height, etc. of the object.
[0026] Compare the 2D projection slice image with the actual acquired 2D image of the positioning target. By calculating the similarity between the images, select the best matching 2D projection slice image and assign it an actual size label. This label may be determined based on the focal length of the camera, the resolution of the image, or the size of the known object. Through this step, the best matching 2D projection slice image can be selected and accurate size information can be provided for it, providing a basis for subsequent precise positioning, ensuring the accuracy and robustness of positioning, and improving the performance of the overall positioning system.
[0027] Step S4: performing geometric positioning according to the acquisition focal length label, the actual size label and the image size label to obtain the image acquisition feature distance.
[0028] Specifically, the image acquisition feature distance refers to the actual distance between the object in the image and the camera, which is calculated through geometry. Read the acquisition focal length label, image size label in step S1, and the actual size label in step S3. Train a neural network model with historical data, input the acquisition focal length label, actual size label, and image size label into the neural network model for geometric positioning, and the model will analyze these labels and output the image acquisition feature distance. Through this step, the actual position and size of the object in the image can be determined to achieve accurate spatial positioning.
[0029] Step S5: using a laser scanner to collect laser point clouds to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identifier.
[0030] Specifically, a laser scanner is a sensor that emits laser light and measures the reflected laser light to create a 3D model of an object or environment. A laser point cloud is a collection of a large number of points generated by a laser scanner that represents the 3D structure of an object or environment. A 3D environment model is a 3D representation of an environment built based on the laser point cloud data, containing all objects in the environment and their spatial relationships. The monocular sensor coordinate identifier specifies the position and orientation of the monocular sensor in the 3D environment model.
[0031] A laser scanner is used to emit a laser and record the time or phase change when the laser is reflected from an object to determine the distance and angle of the object. This data is used to create a laser point cloud, which is a three-dimensional environmental model. When building the model, the position and orientation of the monocular sensor in the environment are recorded, and this information is used as a coordinate identifier. This process can be achieved using laser scanning software such as Rhinoceros or CloudCompare. Through this step, an accurate three-dimensional environmental model can be created that adapts to complex and changing environmental conditions, ensures high-precision positioning under various lighting and texture conditions, and accurately identifies the position of the monocular sensor for use in subsequent positioning and data fusion processes.
[0032] Step S6: According to the monocular sensor coordinate identifier, the image acquisition feature distance locates the positioning target three-dimensional model in the three-dimensional environment model, and performs visual display through the user terminal.
[0033] Specifically, the 3D model of the positioning target is imported into the 3D modeling software, and the position of the 3D model is converted into the coordinate system of the 3D environment model according to the monocular sensor coordinate identification. According to the image acquisition feature distance, the 3D model of the positioning target is accurately placed at the corresponding position in the 3D environment model. Then, it is visualized through the user end so that the user can see the exact position of the target object in the environment. The user end can be a computer screen or a mobile device. This process can be implemented using 3D modeling software (such as Unity or Unreal Engine). Through this step, the target object can be accurately positioned in the 3D environment and visualized through the user end, providing the user with intuitive navigation and positioning information.
[0034] Furthermore, step S2 of the embodiment of the present application further includes:
[0035] Step S21: the first acquisition position tag includes a lens deflection angle tag, wherein the lens deflection angle tag refers to an angle between a monocular sensor lens and a horizontal direction.
[0036] Step S22: setting the lens deflection angle label as image incident angle information, wherein the image incident angle information refers to the angle between a line connecting the first acquisition position and the positioning target and the horizontal direction.
[0037] Step S23: performing two-dimensional projection on the three-dimensional model of the positioning target according to the image incident angle information to obtain the two-dimensional projection slice image.
[0038] Specifically, the lens deflection angle label refers to the angle of the monocular sensor (such as a camera) lens relative to the horizontal direction, describing the degree of lens tilt. It affects the viewing angle of the image and the way objects are projected in the image. Image incident angle information refers to the angle at which light enters the camera lens, that is, the angle between the line from the camera to the target object and the horizontal direction. This angle is crucial to understanding the position and orientation of the object in the image.
[0039] First, the lens deflection angle label is used to determine the tilt angle of the camera. This angle is used as the image incidence angle information, that is, to determine the direction of the camera relative to the target object. Then, this incidence angle information is used to adjust the viewing angle of the 3D model to ensure that the projection of the model is consistent with the viewing angle in the actual image. Next, the adjusted 3D model is perspective projected to generate a 2D projection slice image. This process is usually implemented on a computer using 3D rendering and projection algorithms, such as using 3D modeling software or the projection transformation function in a computer vision library. Through this step, the system can accurately simulate the camera's viewing angle and generate a 2D projection slice image that matches the actual acquired image, providing an accurate reference for subsequent positioning and fusion.
[0040] Further, such as Figure 2 As shown, step S3 of the embodiment of the present application also includes:
[0041] Step S31: Based on the image incident angle information, configure the first alignment corner point and the second alignment corner point of the two-dimensional projection slice image, wherein the first alignment corner point is located at a preset orientation of the first incident position, and the second alignment corner point is located at a preset orientation of the second incident position.
[0042] Step S32: Based on the lens deflection angle label, configure the third alignment corner point and the fourth alignment corner point of the positioning target two-dimensional image, wherein the third alignment corner point is located at a preset orientation of the first incident position, and the fourth alignment corner point is located at a preset orientation of the second incident position.
[0043] Step S33: After aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, shape analysis is performed on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain the selected two-dimensional projection slice image.
[0044] Specifically, the alignment corner points refer to specific points on the image, which are used to align and match two images. The first and second alignment corner points are located on the two-dimensional projection slice image, and the third and fourth alignment corner points are located on the two-dimensional image of the positioning target. The first incident position preset orientation and the second incident position preset orientation refer to specific orientations in the image, which are predefined to help align the images. These orientations may be specific points in the image, such as the corners of a building or the edges of a road sign.
[0045] First, according to the image incident angle information, a specific position on the two-dimensional projection slice image is determined as the first alignment corner point, and another specific position is determined as the second alignment corner point. These corner points are located at a preset position, the position of the first alignment corner point is the preset position of the first incident position, and the position of the second alignment corner point is the preset position of the second incident position. Next, the lens deflection angle label is used to determine the third alignment corner point and the fourth alignment corner point in the two-dimensional image of the positioning target, and these corner points are also located at a preset position. The third alignment corner point is located at the preset position of the first incident position, and the fourth alignment corner point is located at the preset position of the second incident position. Then, the third alignment corner point is aligned with the first alignment corner point, and the fourth alignment corner point is aligned with the second alignment corner point to ensure that the two images are consistent at these key points. This alignment process can be implemented using image registration technology, such as using feature matching algorithms or rigid body transformations to adjust the images so that they match at these points. After alignment, shape analysis is performed to select the best matching two-dimensional projection slice image and assign it an actual size label. The above steps can accurately align and match the images, ensuring that the selected two-dimensional projection slice image is consistent with the actually acquired image at key points, thereby ensuring the consistency of the selected two-dimensional projection slice image with the acquired two-dimensional image of the positioning target.
[0046] Preferably, step S33 of the embodiment of the present application further includes:
[0047] Step S331: after aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, shape similarity analysis is performed on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a shape similarity evaluation result.
[0048] Step S332: When the shape similarity evaluation result is greater than or equal to the shape similarity threshold, the two-dimensional projection slice image is set as the selected two-dimensional projection slice image.
[0049] Step S333: When the shape similarity evaluation result is less than the shape similarity threshold, based on the image incident angle information, two-dimensionally project the three-dimensional model of the positioning target, and update the two-dimensional projection slice image.
[0050] Specifically, shape similarity analysis refers to calculating the similarity score between two images by comparing shape features in the images, such as edges, contours, corners, etc. The shape similarity evaluation result is the output of the shape similarity analysis, usually a numerical value indicating the degree of similarity between the two images. The shape similarity threshold is a pre-set similarity score standard used to determine whether two images are similar enough.
[0051] After aligning the third alignment corner point and the first alignment corner point, as well as the fourth alignment corner point and the second alignment corner point, the shape similarity analysis is performed on the two-dimensional projection slice image and the two-dimensional image of the positioning target to obtain the shape similarity evaluation result. This process can be achieved through edge detection, contour extraction and feature point matching. Then, a similarity threshold is pre-defined, and the shape similarity evaluation result is compared with the shape similarity threshold. If the shape similarity evaluation result is greater than or equal to the preset shape similarity threshold, the current two-dimensional projection slice image is set as the selected two-dimensional projection slice image. If the evaluation result is less than the threshold, indicating that the match is insufficient, the three-dimensional model of the positioning target is re-projected in two dimensions based on the image incident angle information to generate a new two-dimensional projection slice image to find a better match. The two-dimensional projection slice image selected by this step has a high similarity in shape with the actual collected image, thereby ensuring the consistency of the three-dimensional model with the actual scene, providing an accurate reference for subsequent precise positioning, so as to improve the accuracy and robustness of positioning.
[0052] Exemplarily, a car is spatially positioned, and an image of the front view of the car is taken by a monocular sensor. A front view image is projected from a three-dimensional car model. The two-dimensional projection slice image of the car model is aligned with the two-dimensional image of the positioning target according to the alignment corner points, that is, the upper left corner of the car image is aligned with the upper left corner of the projection slice image, and the lower right corner of the car image is aligned with the lower right corner of the projection slice image. Use OpenCV's Canny edge detection algorithm to extract edges in the image. Use the contour detection algorithm to extract contours in the image. Use the SIFT feature point detection and matching algorithm to match feature points. Then, the similarity score is calculated using the number or distance of matched feature points. Set the shape similarity threshold to 0.8. If the similarity score is 0.85, the condition is met and the image is selected.
[0053] Further, such as Figure 3 As shown, step S4 of the embodiment of the present application also includes:
[0054] Step S41: configuring the acquisition of focal length dataset, actual size dataset and image size dataset through the monocular sensor model.
[0055] Step S42: traverse the acquisition focal length dataset, the actual size dataset and the image size dataset to perform sample distance calibration to obtain a historical image acquisition sample distance dataset.
[0056] Step S43: monitor the output of the BP neural network with the historical image acquisition sample distance data set, and configure a feature distance evaluator with the acquisition focal length data set, the actual size data set and the image size data set as input.
[0057] Step S44: geometrically positioning the acquisition focal length label, the actual size label and the image size label according to the feature distance evaluator to obtain the image acquisition feature distance.
[0058] Specifically, the monocular sensor model refers to the specific model of the monocular camera used. The acquisition focal length dataset contains multiple datasets with different focal length values, which record the shooting information under different focal length conditions. The actual size dataset contains multiple datasets of actual sizes of different objects, which are used for calibration and calculation. The image size dataset contains multiple image datasets with different resolutions or sizes. Sample distance calibration refers to determining the actual distance under different parameter combinations through experimental data, which is used to create benchmark data. The historical image acquisition sample distance dataset is a collection that records all distance data obtained through sample distance calibration. The feature distance estimator is a tool that uses neural networks and historical data for prediction and evaluation, and is used to evaluate image acquisition feature distances.
[0059] First, according to the monocular sensor model, configure the acquisition focal length dataset, actual size dataset, and image size dataset. These datasets contain image acquisition samples at different focal lengths and object sizes. Then, traverse these datasets to perform sample distance calibration, that is, use objects with known distances to calibrate the camera to establish the relationship between focal length, actual size, and image size, and generate a historical image acquisition sample distance dataset. Next, use a machine learning framework such as TensorFlow or PyTorch to build a BP neural network model, and use the above sample datasets to train the BP neural network. Use the acquisition focal length dataset, actual size dataset, and image size dataset as model inputs, monitor the output of the BP neural network with the historical image acquisition sample distance dataset, adjust the weights and biases through the back propagation algorithm until the model converges, and configure the trained BP neural network as a feature distance evaluator. This evaluator uses the BP neural network to geometrically locate the acquisition focal length label, actual size label, and image size label, thereby calculating the image acquisition feature distance. The above steps use machine learning methods to improve the accuracy of geometric positioning and ensure that the calculation of image acquisition feature distance is more accurate.
[0060] Preferably, step S42 of the embodiment of the present application further includes:
[0061] Step S421: extracting first acquisition focal length data, first actual size data and first image size data according to the acquisition focal length data set, the actual size data set and the image size data set.
[0062] Step S422: With the first acquired focal length data, the first actual size data and the first image size data as restrictions, historical sample acquisition is performed on the monocular sensor model to obtain a historical sample recorded distance data set.
[0063] Step S423: performing a concentration analysis based on the historical sample record distance data set to obtain first historical image acquisition sample distance data, and adding the data to the historical image acquisition sample distance data set.
[0064] Specifically, a specific focal length value is randomly selected from the acquisition focal length data set as the first acquisition focal length data. A specific actual size data is randomly selected from the actual size data set as the first actual size data. A specific image size data is randomly selected from the image size data set as the first actual size data. Based on the extracted first acquisition focal length data, first actual size data and first image size data, a monocular sensor of a corresponding model is used to perform image acquisition on an object at a known distance to obtain a historical sample recorded distance data set to establish a relationship between focal length, actual size and image size. Among them, the historical sample recorded distance data set is a set of sample distance data collected according to specific conditions. Then, the system performs a central number analysis on the historical sample recorded distance data set, calculates the central tendency of the data, such as the mean or median, obtains the first historical image acquisition sample distance data, and adds it to the historical image acquisition sample distance data set. Repeat the above steps to obtain multiple historical image acquisition sample distance data and add them to the historical image acquisition sample distance data set to provide an accurate data basis for subsequent neural network training.
[0065] Exemplarily, a monocular sensor model of a certain brand is used to collect and analyze sample data for a car model. The following data are extracted from the focal length data set, the actual size data set, and the image size data set: the first focal length data: 50mm; the first actual size data: a 1.5-meter-high car; the first image size data: 1920x1080 pixels. According to the extracted data, the corresponding monocular sensor model is used to collect sample data to obtain the historical sample recorded distance data set under the corresponding focal length, actual size, and measurement distance: {10.1 meters, 10.2 meters, 10.0 meters, 10.3 meters, 10.1 meters}. The historical sample recorded distance data set is analyzed by central number analysis to obtain the first historical image acquisition sample distance data. The average value of the above data set is calculated, and the result is 10.14 meters. The calculated average distance of 10.14 meters is added to the historical image acquisition sample distance data set.
[0066] Furthermore, step S6 of the embodiment of the present application further includes:
[0067] Step S61: performing geometric positioning analysis in the three-dimensional environment model according to the lens deflection angle label, based on the monocular sensor coordinate identifier and the image acquisition feature distance, to obtain the target space coordinates.
[0068] Step S62: obtaining the three-dimensional model posture data of the positioning target according to the acquisition orientation of the selected two-dimensional projection slice image.
[0069] Step S63: using the target space coordinates, deploying the positioning target three-dimensional model in the three-dimensional environment model according to the positioning target three-dimensional model posture data.
[0070] Specifically, the target space coordinates refer to the specific position coordinates of the target object in the three-dimensional environment model. The positioning target three-dimensional model posture data is data describing the specific posture of the positioning target three-dimensional model, such as direction, tilt angle, etc.
[0071] First, according to the lens deflection angle label, the monocular sensor coordinate identification and image acquisition feature distance are used to perform geometric positioning analysis in the three-dimensional environment model. Triangulation or perspective projection technology can be used to obtain the spatial coordinates of the target object. Then, according to the acquisition orientation of the selected two-dimensional projection slice image, the posture data of the target object, that is, the direction and angle of the object, is determined. Finally, the target spatial coordinates and posture data are used to accurately place the three-dimensional model of the positioning target in the three-dimensional environment model.
[0072] For example, the deflection angle of the monocular sensor lens is obtained as -20 degrees, and the sensor coordinates (1, 2, 1.5) and the image acquisition feature distance of 10.14 meters are used to perform geometric positioning analysis, and the target space coordinates are calculated to be (10.613, -1.475, 1.5). Then, according to the selected acquisition orientation of the two-dimensional projection slice image of -20 degrees, the posture data of the three-dimensional model of the positioning target is calculated to be -20 degrees. Finally, the three-dimensional model of the car is positioned to the target space coordinates (10.613, -1.475, 1.5), and rotated -20 degrees according to the calculated posture data, and accurately deployed in the three-dimensional environment model, and visualized through the user end to provide global positioning information.
[0073] In summary, the spatial positioning method of the fusion of monocular vision ranging and laser point cloud provided in the embodiment of the present application has the following technical effects:
[0074] A two-dimensional image of a positioning target is received through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag, and an image size tag. These tags provide the shooting position of the image, the camera focal length, and the size information of the image, and provide basic data for subsequent positioning calculations. The three-dimensional model of the positioning target is loaded, and two-dimensional projection is performed according to the first acquisition position tag to obtain a two-dimensional projection slice image; this step converts the three-dimensional information into a two-dimensional image, and generates a slice image that matches the actual scene, so as to be compared with the image collected by the monocular sensor. The two-dimensional projection slice image and the two-dimensional image of the positioning target are subjected to shape analysis to obtain a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size tag. This step ensures that the selected two-dimensional projection slice image accurately matches the actual image through shape analysis, provides reliable size information, and provides an accurate data basis for geometric positioning. Geometric positioning is performed according to the acquisition focal length tag, the actual size tag, and the image size tag to obtain the image acquisition feature distance. This step accurately calculates the distance information of the target object in the actual space by combining the focal length and image size information, ensures the accuracy of the depth information, and improves the positioning accuracy. Through the laser scanner, the laser point cloud is collected to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identification. This step constructs an accurate three-dimensional environment model, provides rich spatial information and depth data, enhances the system's environmental perception ability, and can adapt to complex and changing environmental conditions. According to the monocular sensor coordinate identification, the image acquisition feature distance locates the positioning target three-dimensional model in the three-dimensional environment model, and visualizes it through the user end. This step calculates the posture data of the target three-dimensional model through geometric positioning analysis and the acquisition orientation of the two-dimensional projection slice image, realizes the precise placement and posture adjustment of the target object in the three-dimensional environment, and ensures the authenticity and accuracy of the positioning; and through visual display, the user can intuitively see the positioning results.
[0075] In general, the embodiment of the present application combines monocular vision ranging and laser point cloud technology, and the three-dimensional environment model constructed by the laser scanner provides accurate three-dimensional spatial information. At the same time, the three-dimensional model of the positioning target is used, and compared with the actual captured two-dimensional image through two-dimensional projection, and geometric positioning is performed using the acquisition focal length, actual size and image size. The characteristic distance of image acquisition is calculated more accurately, so that the positioning target can be better integrated with the actual environment, and the accuracy of target recognition and positioning is improved, thereby significantly improving the accuracy and reliability of spatial positioning, and being able to better adapt to complex or changing environments and enhancing environmental adaptability.
[0076] Embodiment 2, as Figure 4As shown, the embodiment of the present application provides a spatial positioning system integrating monocular visual ranging and laser point cloud, and the system includes:
[0077] The monocular ranging module 10 is used to receive a two-dimensional image of a positioning target through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag and an image size tag.
[0078] The two-dimensional projection module 20 is used to load the three-dimensional model of the positioning target, perform two-dimensional projection according to the first acquisition position tag, and obtain a two-dimensional projection slice image.
[0079] The shape analysis module 30 is used to perform shape analysis on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size label.
[0080] The geometric positioning module 40 is used to perform geometric positioning according to the acquisition focal length label, the actual size label and the image size label to obtain the image acquisition feature distance.
[0081] The three-dimensional environment model construction module 50 is used to collect laser point clouds through a laser scanner to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identifier.
[0082] The target positioning module 60 is used to position the target three-dimensional model in the three-dimensional environment model according to the monocular sensor coordinate identification and the image acquisition feature distance, and to perform visual display through the user terminal.
[0083] Furthermore, the two-dimensional projection module 20 in the embodiment of the present application is also used to perform the following steps:
[0084] The first acquisition position tag includes a lens deflection angle tag, wherein the lens deflection angle tag refers to the angle between the monocular sensor lens and the horizontal direction; the lens deflection angle tag is set to image incident angle information, wherein the image incident angle information refers to the angle between the line connecting the first acquisition position and the positioning target and the horizontal direction; according to the image incident angle information, the three-dimensional model of the positioning target is two-dimensionally projected to obtain the two-dimensional projection slice image.
[0085] Furthermore, the shape analysis module 30 in the embodiment of the present application is also used to perform the following steps:
[0086] Based on the image incident angle information, a first alignment corner point and a second alignment corner point of the two-dimensional projection slice image are configured, wherein the first alignment corner point is located at a preset position of the first incident position, and the second alignment corner point is located at a preset position of the second incident position; based on the lens deflection angle label, a third alignment corner point and a fourth alignment corner point of the positioning target two-dimensional image are configured, wherein the third alignment corner point is located at a preset position of the first incident position, and the fourth alignment corner point is located at a preset position of the second incident position; after aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, shape analysis is performed on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain the selected two-dimensional projection slice image.
[0087] Furthermore, the shape analysis module 30 in the embodiment of the present application is also used to perform the following steps:
[0088] After aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, shape similarity analysis is performed on the two-dimensional projection slice image and the two-dimensional image of the positioning target to obtain a shape similarity evaluation result; when the shape similarity evaluation result is greater than or equal to a shape similarity threshold, the two-dimensional projection slice image is set as the selected two-dimensional projection slice image; when the shape similarity evaluation result is less than the shape similarity threshold, based on the image incident angle information, two-dimensional projection is performed on the three-dimensional model of the positioning target, and the two-dimensional projection slice image is updated.
[0089] Furthermore, the geometric positioning module 40 in the embodiment of the present application is also used to perform the following steps:
[0090] Through the monocular sensor model, the acquisition focal length dataset, the actual size dataset and the image size dataset are configured; the acquisition focal length dataset, the actual size dataset and the image size dataset are traversed to perform sample distance calibration to obtain a historical image acquisition sample distance dataset; the output of the BP neural network is monitored with the historical image acquisition sample distance dataset, and a feature distance evaluator is configured with the acquisition focal length dataset, the actual size dataset and the image size dataset as input; the acquisition focal length label, the actual size label and the image size label are geometrically positioned according to the feature distance evaluator to obtain the image acquisition feature distance.
[0091] Furthermore, the geometric positioning module 40 in the embodiment of the present application is also used to perform the following steps:
[0092] According to the acquisition focal length data set, the actual size data set and the image size data set, first acquisition focal length data, first actual size data and first image size data are extracted; with the first acquisition focal length data, the first actual size data and the first image size data as restrictions, historical sample acquisition is performed on the monocular sensor model to obtain a historical sample recorded distance data set; based on the historical sample recorded distance data set, a centralized number analysis is performed to obtain first historical image acquisition sample distance data, and the data is added to the historical image acquisition sample distance data set.
[0093] Furthermore, the target positioning module 60 in the embodiment of the present application is also used to perform the following steps:
[0094] According to the lens deflection angle label, a geometric positioning analysis is performed in the three-dimensional environment model based on the monocular sensor coordinate identifier and the image acquisition feature distance to obtain the target space coordinates; according to the acquisition orientation of the selected two-dimensional projection slice image, the three-dimensional model posture data of the positioning target is obtained; with the target space coordinates, the three-dimensional model of the positioning target is deployed in the three-dimensional environment model according to the three-dimensional model posture data of the positioning target.
[0095] Through the above-mentioned detailed description of the spatial positioning method of the fusion of monocular vision ranging and laser point cloud in this specification, those skilled in the art can clearly understand the spatial positioning system of the fusion of monocular vision ranging and laser point cloud in this embodiment. For the system disclosed in Example 2, since it corresponds to the method disclosed in Example 1, it has corresponding functional modules and beneficial effects. For relevant matters, please refer to the description of the method part.
[0096] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A spatial positioning method combining monocular vision ranging and laser point cloud, characterized in that: include: Receive a two-dimensional image of a positioning target through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag, and an image size tag, wherein the first acquisition position tag is a specific position of the monocular sensor when recording the image, the acquisition focal length tag refers to the focal length of the lens used when the image is captured, and the image size tag is a resolution or size of the recorded image; Loading the three-dimensional model of the positioning target, performing two-dimensional projection according to the first acquisition position label, and obtaining a two-dimensional projection slice image; Performing shape analysis on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size label, and the actual size label refers to actual size information of an object in the image; Performing geometric positioning according to the acquisition focal length tag, the actual size tag and the image size tag to obtain an image acquisition feature distance, wherein the image acquisition feature distance refers to the actual distance between the object in the image and the camera; By means of a laser scanner, a laser point cloud is collected to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identifier, and the monocular sensor coordinate identifier specifies a position and an orientation of the monocular sensor in the three-dimensional environment model; According to the monocular sensor coordinate identifier, the image acquisition feature distance positions the positioning target three-dimensional model on the three-dimensional environment model, and performs visual display through the user terminal.
2. The spatial positioning method of monocular vision ranging and laser point cloud fusion as claimed in claim 1, characterized in that: Loading the three-dimensional model of the positioning target, performing two-dimensional projection according to the first acquisition position tag, and obtaining a two-dimensional projection slice image, including: The first acquisition position tag includes a lens deflection angle tag, wherein the lens deflection angle tag refers to the angle between the monocular sensor lens and the horizontal direction; The lens deflection angle label is set as image incident angle information, wherein the image incident angle information refers to the angle between the line connecting the first acquisition position and the positioning target and the horizontal direction; According to the image incident angle information, the three-dimensional model of the positioning target is two-dimensionally projected to obtain the two-dimensional projection slice image.
3. The spatial positioning method of monocular vision ranging and laser point cloud fusion as claimed in claim 2, characterized in that: Performing shape analysis on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a selected two-dimensional projection slice image includes: Based on the image incident angle information, configuring a first alignment corner point and a second alignment corner point of the two-dimensional projection slice image, wherein the first alignment corner point is located at a preset orientation of the first incident position, and the second alignment corner point is located at a preset orientation of the second incident position; Based on the lens deflection angle label, configuring a third alignment corner point and a fourth alignment corner point of the two-dimensional image of the positioning target, wherein the third alignment corner point is located at a preset orientation of the first incident position, and the fourth alignment corner point is located at a preset orientation of the second incident position; After aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, shape analysis is performed on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain the selected two-dimensional projection slice image.
4. The spatial positioning method of monocular vision ranging and laser point cloud fusion as claimed in claim 3 is characterized in that: After aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, shape analysis is performed on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain the selected two-dimensional projection slice image, including: After aligning the third alignment corner point with the first alignment corner point, and aligning the fourth alignment corner point with the second alignment corner point, performing shape similarity analysis on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a shape similarity evaluation result; When the shape similarity evaluation result is greater than or equal to the shape similarity threshold, setting the two-dimensional projection slice image as the selected two-dimensional projection slice image; When the shape similarity evaluation result is less than the shape similarity threshold, based on the image incident angle information, two-dimensional projection is performed on the three-dimensional model of the positioning target, and the two-dimensional projection slice image is updated.
5. The spatial positioning method of monocular vision ranging and laser point cloud fusion as claimed in claim 1, characterized in that: Performing geometric positioning according to the acquisition focal length label, the actual size label, and the image size label to obtain an image acquisition feature distance includes: Through the monocular sensor model, configure the collection of focal length dataset, actual size dataset and image size dataset; Traversing the acquisition focal length dataset, the actual size dataset and the image size dataset to perform sample distance calibration, and obtain a historical image acquisition sample distance dataset; The output of the BP neural network is monitored using the historical image acquisition sample distance data set, and a feature distance evaluator is configured using the acquisition focal length data set, the actual size data set and the image size data set as inputs; The acquisition focal length label, the actual size label and the image size label are geometrically positioned according to the feature distance evaluator to obtain the image acquisition feature distance.
6. The spatial positioning method of monocular vision ranging and laser point cloud fusion as claimed in claim 5, characterized in that: Traversing the acquisition focal length dataset, the actual size dataset, and the image size dataset to perform sample distance calibration to obtain a historical image acquisition sample distance dataset, including: Extracting first acquisition focal length data, first actual size data and first image size data according to the acquisition focal length data set, the actual size data set and the image size data set; Taking the first acquisition focal length data, the first actual size data and the first image size data as restrictions, historical sample collection is performed on the monocular sensor model to obtain a historical sample recorded distance data set; A centralized number analysis is performed on the historical sample record distance data set to obtain first historical image acquisition sample distance data, which is added to the historical image acquisition sample distance data set.
7. The spatial positioning method of combining monocular vision ranging with laser point cloud as claimed in claim 2, characterized in that: According to the monocular sensor coordinate identifier, the image acquisition feature distance locates the positioning target three-dimensional model on the three-dimensional environment model, including: According to the lens deflection angle label, based on the monocular sensor coordinate identifier and the image acquisition feature distance, a geometric positioning analysis is performed on the three-dimensional environment model to obtain the target space coordinates; Obtaining the three-dimensional model posture data of the positioning target according to the acquisition orientation of the selected two-dimensional projection slice image; Using the target space coordinates, the positioning target three-dimensional model is deployed in the three-dimensional environment model according to the positioning target three-dimensional model posture data.
8. The spatial positioning system integrating monocular vision ranging and laser point cloud is characterized by: The system is used to execute the spatial positioning method of fusion of monocular visual ranging and laser point cloud according to any one of claims 1 to 7, comprising: A monocular ranging module, wherein the monocular ranging module is used to receive a two-dimensional image of a positioning target through a monocular sensor, wherein the two-dimensional image of the positioning target has a first acquisition position tag, an acquisition focal length tag, and an image size tag, wherein the first acquisition position tag is a specific position of the monocular sensor when recording the image, the acquisition focal length tag refers to the focal length of the lens used when the image is captured, and the image size tag is a resolution or size of the recorded image; A two-dimensional projection module, the two-dimensional projection module is used to load the three-dimensional model of the positioning target, perform two-dimensional projection according to the first acquisition position tag, and obtain a two-dimensional projection slice image; A shape analysis module, the shape analysis module is used to perform shape analysis on the two-dimensional projection slice image and the positioning target two-dimensional image to obtain a selected two-dimensional projection slice image, wherein the selected two-dimensional projection slice image has an actual size label, and the actual size label refers to the actual size information of the object in the image; A geometric positioning module, the geometric positioning module is used to perform geometric positioning according to the acquisition focal length label, the actual size label and the image size label to obtain an image acquisition feature distance, wherein the image acquisition feature distance refers to the actual distance between the object in the image and the camera; A three-dimensional environment model construction module, wherein the three-dimensional environment model construction module is used to collect laser point clouds through a laser scanner to construct a three-dimensional environment model, wherein the three-dimensional environment model has a monocular sensor coordinate identifier, and the monocular sensor coordinate identifier specifies the position and direction of the monocular sensor in the three-dimensional environment model; The target positioning module is used to position the three-dimensional model of the positioning target in the three-dimensional environment model according to the coordinate identification of the monocular sensor and the image acquisition feature distance, and to perform visual display through the user terminal.
