Sandbox positioning method based on 3D vision

Abstract

The invention relates to the technical field of vision and positioning, and discloses a sandbox positioning method based on 3D vision, and the method comprises the steps: scanning a sandbox, and obtaining an original three-dimensional point cloud of an object in the sandbox; preprocessing the collected original three-dimensional point cloud to obtain an ordered three-dimensional point cloud; performing feature extraction and matching on the ordered three-dimensional point cloud; preliminarily positioning the object according to a matching result; and optimizing the preliminary positioning result. According to the invention, the positioning precision is greatly improved.

Description

Technical Field

[0001] The present invention relates to the fields of vision technology and positioning technology, and specifically to a method for sandbox positioning based on 3D vision. Background Art

[0002] With the rapid development of computer vision and sensor technology, 3D vision technology has been increasingly widely used in multiple fields. Especially in fields such as sandbox games, robot navigation, and terrain mapping, the precise positioning of objects has become a core requirement. Due to its inherent limitations, traditional 2D vision technology is difficult to meet the requirements for high-precision positioning in these fields. The deficiencies of the existing technology include:

[0003] (1) Two-dimensional limitation: Traditional 2D vision technology can only provide the planar projection information of an object and cannot directly obtain the spatial position and attitude of the object.

[0004] (2) Inaccurate positioning: In complex natural scenes, it is difficult to accurately extract object features from 2D images for matching, resulting in insufficient positioning accuracy.

[0005] Therefore, a sandbox positioning method based on 3D vision has emerged, aiming to achieve precise identification and positioning of objects by obtaining the three-dimensional information of objects in the sandbox and combining advanced data processing technologies and algorithm analysis. Summary of the Invention

[0006] (I) Technical problems to be solved

[0007] In view of the deficiencies of the existing technology, the present application provides a method for sandbox positioning based on 3D vision.

[0008] (II) Technical solutions

[0009] To solve the above problems, the present application provides the following technical solutions:

[0010] A method for sandbox positioning based on 3D vision, comprising:

[0011] Scanning the sandbox to obtain the original three-dimensional point cloud of the objects in the sandbox;

[0012] Preprocessing the collected original three-dimensional point cloud to obtain an ordered three-dimensional point cloud;

[0013] Performing feature extraction and matching on the ordered three-dimensional point cloud;

[0014] Performing preliminary positioning on the object according to the matching result;

[0015] Optimizing the preliminary positioning result.

[0016] Preferably, the step of scanning the sandbox to obtain the original three-dimensional point cloud of the object in the sandbox specifically includes:

[0017] Perform three-dimensional scanning on the sandbox and the objects inside it through a 3D scanning device to obtain the original three-dimensional point cloud of the objects in the sandbox. The data information of the three-dimensional point cloud includes at least the shape, size, and position of the objects.

[0018] Preferably, the step of extracting and matching features from the ordered three-dimensional point cloud specifically includes:

[0019] Extract features from the ordered three-dimensional point cloud. The extracted feature information includes at least the edges, planes, corner points, and textures of the objects.

[0020] Find the positions where the depth values change abruptly in the ordered three-dimensional point cloud to determine the edges of the objects; detect the feature information of the planes on the object surfaces by fitting planes; determine the feature information of the corner points according to the intersection lines and angular relationships between the planes; extract the texture feature information of the objects through the gray-level co-occurrence matrix.

[0021] Match the extracted feature information with a pre-established model library for data.

[0022] For the edge and corner point features, calculate the Hausdorff distance between the edge point set of the sandbox objects and the edge model point set of the objects in the model library to measure the similarity of the features. When the distance value is less than the set threshold, there is similarity.

[0023] For the texture features, calculate the cosine similarity between the texture feature vectors of the sandbox objects and the texture feature vectors of each model in the model library. The closer the cosine value is to 1, the more similar the texture features of the two are.

[0024] Preferably, the step of performing preliminary positioning on the object according to the matching result specifically includes:

[0025] Perform positioning based on the object features. Use the feature points on the object surface for positioning. According to the feature matching result, calculate the relative position relationship between the feature points to determine the position of the object in the sandbox.

[0026] Perform positioning based on the object model. Use the pre-established object model for positioning. By comparing the differences in shape, size, and angle between the model and the actual object, calculate the accurate position of the object in the sandbox.

[0027] Preferably, the step of optimizing the preliminary positioning result specifically includes:

[0028] Construct the positioning neural network; determine the input layer, convolutional layer, pooling layer, fully connected layer, and output layer of the network structure.

[0029] Train the positioning neural network.

[0030] Evaluate the model; use the test set to evaluate the trained model, and calculate the mean and standard deviation of the position error and the attitude error;

[0031] Update the positioning neural network; according to the feedback of the output result, retrain the positioning neural network and update the network structure.

[0032] Preferably, constructing the positioning neural network specifically includes:

[0033] a. The input layer receives the preliminary positioning result and the ordered three-dimensional point cloud, and normalizes the received data information, scaling the coordinate values to the range of [0, 1]; the preliminary positioning result includes the preliminary spatial coordinates (X0, Y0, Z0) and the preliminary attitude angles (α0, β0, γ0), X0 is the preliminary spatial abscissa, Y0 is the preliminary spatial ordinate, Z0 is the preliminary spatial height coordinate, α0 is the preliminary spatial yaw angle, β0 is the preliminary spatial pitch angle, and γ0 is the preliminary spatial roll angle;

[0034] b. The convolutional layer includes 5 convolutional layers. The first convolutional layer includes 32 convolutional kernels, the second convolutional layer includes 64 convolutional kernels, the third convolutional layer includes 64 convolutional kernels, the fourth convolutional layer includes 32 convolutional kernels, and the fifth convolutional layer includes 16 convolutional kernels; the size of the convolutional kernel is 3×3×3;

[0035] c. The pooling layer is used to reduce the dimension of the data; the pooling layer is inserted between every two convolutional layers, and the size of the pooling layer is 2×2×2;

[0036] d. The fully connected layer includes the first fully connected layer and the second fully connected layer; the feature map processed by the convolutional layer and the pooling layer is flattened into a one-dimensional vector and connected to the first fully connected layer, and the first fully connected layer outputs to the second fully connected layer; the number of neurons in the second fully connected layer is 6;

[0037] e. The output layer outputs the final positioning result, the spatial coordinates (X, Y, Z) and the attitude angles (α, β, γ); the output layer includes 6 neurons, X is the spatial abscissa, Y is the spatial ordinate, Z is the spatial height coordinate, ɑ is the spatial yaw angle, β is the spatial pitch angle, and γ is the spatial roll angle.

[0038] Preferably, the training of the positioning neural network specifically includes:

[0039] a. Train the input data in batches, and each batch contains 128 samples;

[0040] b. Quantify the difference between the model prediction value and the true value through the loss function;

[0041] The mathematical expression of the position loss function is as follows:

[0042]

[0043] In formula (1), E p is the position loss error, X i , Y i , Z i are the true position coordinates, and X′ i , Y′ i , Z′ i are the predicted position coordinates;

[0044] The mathematical expression of the attitude loss function is as follows:

[0045]

[0046] In formula (2), E q is the attitude loss error, ɑ i , β i , γ i are the true angles, and ɑ′ i , β′ i , γ′ i are the predicted angles;

[0047] The mathematical expression of the comprehensive loss function is as follows:

[0048] E = w1E p + w2E q + σ|E p - E q | (3)

[0049] In formula (3), w1 is the position weight, w2 is the attitude weight, σ is the balance weight, and E is the comprehensive loss error;

[0050] c. Calculate the gradient of the loss function with respect to the parameters in the network through backpropagation;

[0051] d. Calculate the new parameter values through the Adam optimizer according to its update rule using the gradient and the current parameter values, and update the parameter values to move the parameters of the network in the direction of reducing the loss function;

[0052] e. Repeat the above steps until the training of all batches of data is completed.

[0053] (III) Beneficial Effects

[0054] Compared with the prior art, the present application provides a method for sandbox positioning based on 3D vision, which has the following beneficial effects:

[0055] 1. This method extracts features from point clouds through 3D vision technology, which can accurately capture the key features of objects and provide precise basic data for positioning. Whether the object has a regular shape or complex surface features, feature extraction and positioning can be performed through point cloud data.

[0056] 2. This method performs preliminary positioning by combining the object feature positioning algorithm and the object model positioning algorithm. It can utilize the surface feature points of the object and the pre-established object model to comprehensively calculate the position of the object, improving the accuracy of positioning. The positioning neural network further optimizes the preliminary positioning result using its powerful feature learning and data fitting capabilities, effectively reducing errors and achieving high-precision positioning.

[0057] 3. This method continuously improves the positioning performance through the self-learning and adaptive capabilities of the positioning neural network, automatically learning and optimizing the positioning strategy for different scenarios.

[0058] Additional aspects and advantages of the present application will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0060] Figure 1 is the overall architecture diagram of a method for sandbox positioning based on 3D vision according to the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0062] The terms "first" and "second" in the description and claims of the present application may explicitly or implicitly include one or more of such features. In the description of the present application, unless otherwise stated, the meaning of "a plurality" is two or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0063] The present application provides a new technical solution:

[0064] A method for sandbox positioning based on 3D vision, comprising:

[0065] Scan the sandbox to obtain the original three-dimensional point cloud of the objects inside the sandbox;

[0066] Preprocess the collected original three-dimensional point cloud to obtain an ordered three-dimensional point cloud;

[0067] Extract and match features from the ordered three-dimensional point cloud;

[0068] Perform a preliminary positioning of the object based on the matching result;

[0069] Optimize the preliminary positioning result.

[0070] In the present invention, the step of scanning the sandbox to obtain the original three-dimensional point cloud of the objects inside the sandbox specifically includes:

[0071] Perform three-dimensional scanning on the sandbox and the objects inside it through a 3D scanning device to obtain the original three-dimensional point cloud of the objects inside the sandbox. The data information of the three-dimensional point cloud includes at least the shape, size, and position of the objects.

[0072] In a specific embodiment, the 3D scanning device can be a 3D vision sensor.

[0073] In the present invention, the step of extracting and matching features from the ordered three-dimensional point cloud specifically includes:

[0074] Extract features from the ordered three-dimensional point cloud. The extracted feature information includes at least the edges, planes, corner points, and textures of the objects;

[0075] Find the positions where the depth values change abruptly in the ordered three-dimensional point cloud to determine the edges of the objects; detect the feature information of the plane on the object surface by fitting a plane; determine the feature information of the corner points according to the intersection line and angular relationship between the planes; extract the texture feature information of the objects through a gray-level co-occurrence matrix;

[0076] Match the extracted feature information with a pre-established model library for data;

[0077] a. For the edge and corner point features, measure the similarity of the features by calculating the Hausdorff distance between the edge point set of the sandbox objects and the edge model point set of the objects in the model library. When the distance value is less than the set threshold, there is similarity;

[0078] b. For the texture features, calculate the cosine similarity between the texture feature vector of the sandbox objects and the texture feature vectors of each model in the model library. The closer the cosine value is to 1, the more similar the texture features of the two are.

[0079] In a specific embodiment, the position where the depth value changes abruptly is found in the ordered three-dimensional point cloud through Canny edge detection to determine the edge of the object; for Canny edge detection, the Gaussian kernel size is selected as 5×5, the standard deviation is 1.4, the gradient magnitude and direction of the image are calculated, the edge is refined through non-maximum suppression, the depth change threshold is set to 0.01 m, and the edge is determined according to the depth value change; the projected points in the ordered three-dimensional point cloud are traversed, and if the depth value change between a projected point and its neighboring projected points exceeds the set threshold and the projected point is retained in the non-maximum suppression step, then this point is considered to be located on the object edge.

[0080] The texture feature information of the object is extracted through the gray-level co-occurrence matrix. The ordered three-dimensional point cloud is projected onto a two-dimensional plane, and a gray-scale image is generated according to the reflection intensity of the points. The gray-level co-occurrence matrix is calculated, the gray level is set to 8, and the gray-level co-occurrence matrices in different directions 0, 45, 90, 135 and different distances 1, 2, 3 are calculated.

[0081] In the present invention, the preliminary positioning of the object according to the matching result specifically includes:

[0082] Positioning based on object features, using the feature points on the object surface for positioning. According to the feature matching result, the relative position relationship between the feature points is calculated to determine the position of the object in the sandbox;

[0083] Positioning based on the object model, using the pre-established object model for positioning. By comparing the differences in shape, size and angle between the model and the actual object, the precise position of the object in the sandbox is calculated.

[0084] In a specific embodiment, for the positioning based on the object model, an accurate three-dimensional model of the object is established. For each model, its shape, size and the relative position relationship between its parts are defined. Feature extraction is performed on the established object model, and the extracted features include edges, planes and corner points;

[0085] By comparing the edge and plane features of the actual object and the model, the shape difference is calculated;

[0086] According to the feature matching result, the distance ratio relationship between the corresponding feature points in the actual object and the model is determined, and the size difference is calculated;

[0087] By comparing the included angle between the planes in the actual object and the model, as well as the overall rotation angle of the object, the angle difference is calculated.

[0088] In the present invention, the optimization of the preliminary positioning result specifically includes:

[0089] Construct the positioning neural network; determine the input layer, convolutional layer, pooling layer, fully connected layer and output layer of the network structure;

[0090] Train the positioning neural network;

[0091] Evaluate the model; Use the test set to evaluate the trained model, and calculate the average and standard deviation of the position error and the attitude error;

[0092] Update the positioning neural network; According to the feedback of the output result, retrain the positioning neural network and update the network structure.

[0093] In the present invention, the construction of the positioning neural network specifically includes:

[0094] a. The input layer receives the result of the preliminary positioning and the ordered three-dimensional point cloud, and normalizes the received data information, scaling the coordinate values to the range of [0, 1]; The preliminary positioning result includes the preliminary spatial coordinates (X0, Y0, Z0) and the preliminary attitude angles (α0, β0, γ0), where X0 is the preliminary spatial abscissa, Y0 is the preliminary spatial ordinate, Z0 is the preliminary spatial height coordinate, α0 is the preliminary spatial yaw angle, β0 is the preliminary spatial pitch angle, and γ0 is the preliminary spatial roll angle;

[0095] b. The convolutional layer includes 5 convolutional layers. The first convolutional layer includes 32 convolutional kernels, the second convolutional layer includes 64 convolutional kernels, the third convolutional layer includes 64 convolutional kernels, the fourth convolutional layer includes 32 convolutional kernels, and the fifth convolutional layer includes 16 convolutional kernels; The size of the convolutional kernel is 3×3×3;

[0096] c. The pooling layer is used to reduce the dimension of the data; The pooling layer is inserted between every two convolutional layers, and the size of the pooling layer is 2×2×2;

[0097] d. The fully connected layer includes the first fully connected layer and the second fully connected layer; The feature map processed by the convolutional layer and the pooling layer is flattened into a one-dimensional vector and connected to the first fully connected layer, and the first fully connected layer outputs to the second fully connected layer; The number of neurons in the second fully connected layer is 6;

[0098] e. The output layer outputs the final positioning result, the spatial coordinates (X, Y, Z) and the attitude angles (α, β, γ); The output layer includes 6 neurons, where X is the spatial abscissa, Y is the spatial ordinate, Z is the spatial height coordinate, α is the spatial yaw angle, β is the spatial pitch angle, and γ is the spatial roll angle.

[0099] In the present invention, the training of the positioning neural network specifically includes:

[0100] a. Train the input data in batches, and each batch contains 128 samples;

[0101] b. Quantify the difference between the predicted value and the true value of the model through the loss function;

[0102] The mathematical expression of the position loss function is:

[0103]

[0104] In formula (1), E p is the position loss error, X i , Y i , Z i are the true position coordinates, X′ i , Y′ i , z′ i are the predicted position coordinates;

[0105] The mathematical expression of the attitude loss function is:

[0106]

[0107] In formula (2), E q is the attitude loss error, α i , β i , γ i are the true angles, α′ i , β′ i , γ′ i are the predicted angles;

[0108] The mathematical expression of the comprehensive loss function is:

[0109] E = w1E p + w2E q + σ|E p - E q | (3)

[0110] In formula (3), w1 is the position weight, w2 is the attitude weight, σ is the balance weight, and E is the comprehensive loss error;

[0111] c. Calculate the gradient of the loss function with respect to the parameters in the network through backpropagation;

[0112] d. Calculate the new parameter values through the Adam optimizer according to its update rule using the gradient and the current parameter values, and update the parameter values to move the parameters of the network in the direction that reduces the loss function;

[0113] e. Repeat the above steps until the training of all batches of data is completed.

[0114] In a specific embodiment, w1 is the position weight, w2 is the attitude weight, and σ is the balance weight, which can be adaptively adjusted according to the accuracy of the positioning result; the position weight w1 is set to be greater than 0.7 to increase the accuracy of position prediction and reduce the position error; the attitude weight w2 is set to be greater than 0.6 to reduce the attitude loss; the balance weight σ is used to balance the difference between the position loss and the attitude loss to prevent either side from dominating absolutely in the comprehensive loss. When the magnitude difference between the position loss and the attitude loss is greater than 10, the balance weight σ is set to be greater than 0.5. When the magnitude difference between the position loss and the attitude loss is 0.1 - 10, the balance weight σ is set to 0.3 - 0.5.

[0115] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0116] Although the embodiments of the present application have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application. The scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for sandbox positioning based on 3D vision, characterized in that, Including: Scanning the sandbox to obtain the original three-dimensional point cloud of the object inside the sandbox; Preprocessing the collected original three-dimensional point cloud to obtain an ordered three-dimensional point cloud; Performing feature extraction and matching on the ordered three-dimensional point cloud; Performing preliminary positioning on the object according to the matching result; Optimizing the preliminary positioning result.

2. The method for sandbox positioning based on 3D vision according to claim 1, wherein The scanning of the sandbox to obtain the original three-dimensional point cloud of the object inside the sandbox specifically includes: Performing three-dimensional scanning on the sandbox and the object inside it through a 3D scanning device to obtain the original three-dimensional point cloud of the object inside the sandbox, and the data information of the three-dimensional point cloud at least includes the shape, size and position of the object.

3. A method for sandbox positioning based on 3D vision according to claim 1, characterized in that, The performing feature extraction and matching on the ordered three-dimensional point cloud specifically includes: Performing feature extraction on the ordered three-dimensional point cloud, and the extracted feature information at least includes the edges, planes, corner points and textures of the object; Finding the positions where the depth values change suddenly in the ordered three-dimensional point cloud to determine the edges of the object; detecting the feature information of the plane on the object surface by fitting a plane; determining the feature information of the corner points according to the intersection line and angle relationship between the planes; extracting the texture feature information of the object through a gray-level co-occurrence matrix; Performing data matching on the extracted feature information with a pre-established model library; a. For the edge and corner point features, by calculating the Hausdorff distance between the edge point set of the sandbox object and the edge model point set of the object in the model library, the similarity of the features is measured. When the distance value is less than the set threshold, there is similarity; b. For the texture features, calculating the cosine similarity between the texture feature vector of the sandbox object and the texture feature vectors of each model in the model library. The closer the cosine value is to 1, the more similar the two texture features are.

4. A method for sandbox positioning based on 3D vision according to claim 1, characterized in that, The performing preliminary positioning on the object according to the matching result specifically includes: Performing positioning based on object features, using the feature points on the object surface for positioning. According to the feature matching result, calculating the relative position relationship between the feature points to determine the position of the object in the sandbox; Performing positioning based on the object model, using a pre-established object model for positioning. By comparing the differences in shape, size and angle between the model and the actual object, calculating the accurate position of the object in the sandbox.

5. A method for sandbox positioning based on 3D vision according to claim 1, characterized in that, The optimizing the preliminary positioning result specifically includes: Constructing the positioning neural network; determining the input layer, convolutional layer, pooling layer, fully connected layer and output layer of the network structure; Training the positioning neural network; Evaluating the model; evaluating the trained model using a test set, and calculating the average value and standard deviation of the position error and attitude error; Updating the positioning neural network; according to the feedback of the output result, retraining the positioning neural network and updating the network structure.

6. A method for sandbox positioning based on three-dimensional vision according to claim 5, wherein The constructing the positioning neural network specifically includes: a. The input layer receives the result of preliminary positioning and the ordered three-dimensional point cloud, and normalizes the received data information, scaling the coordinate values to the range of [0, 1]; the preliminary positioning result includes the preliminary spatial coordinates (X0, Y0, Z0) and the preliminary attitude angles (α0, β0, γ0), where X0 is the preliminary spatial abscissa, Y0 is the preliminary spatial ordinate, Z0 is the preliminary spatial height coordinate, α0 is the preliminary spatial yaw angle, β0 is the preliminary spatial pitch angle, and γ0 is the preliminary spatial roll angle; b. The convolutional layer includes 5 convolutional layers. The first convolutional layer includes 32 convolutional kernels, the second convolutional layer includes 64 convolutional kernels, the third convolutional layer includes 64 convolutional kernels, the fourth convolutional layer includes 32 convolutional kernels, and the fifth convolutional layer includes 16 convolutional kernels; the size of the convolutional kernel is 3×3×3; c. The pooling layer is used to reduce the dimension of the data; the pooling layer is inserted between every two convolutional layers, and the size of the pooling layer is 2×2×2; d. The fully connected layer includes the first fully connected layer and the second fully connected layer; the feature map processed by the convolutional layer and the pooling layer is flattened into a one-dimensional vector and connected to the first fully connected layer, and the first fully connected layer outputs to the second fully connected layer; the number of neurons in the second fully connected layer is 6; e. The output layer outputs the final positioning result, the spatial coordinates (X, Y, Z) and the attitude angles (α, β, γ); the output layer includes 6 neurons, where X is the spatial abscissa, Y is the spatial ordinate, Z is the spatial height coordinate, α is the spatial yaw angle, β is the spatial pitch angle, and γ is the spatial roll angle.

7. A method for sandbox positioning based on 3D vision according to claim 5, characterized in that, Training the positioning neural network specifically includes: a. Training the input data in batches, with each batch containing 128 samples; b. Quantifying the difference between the model prediction value and the true value through the loss function; The mathematical expression of the position loss function is: In formula (1), E p is the position loss error, X i , Y i , Z i are the true position coordinates, X′ i , Y′ i , Z′ i are the predicted position coordinates; The mathematical expression of the attitude loss function is: In formula (2), E q is the attitude loss error, ɑ i , β i , γ i are the true angles, α′ i , β′ i , γ′ i are the predicted angles; The mathematical expression of the comprehensive loss function is: E = w1E p + w2E q + σ|E p - E q | (3) In formula (3), w1 is the position weight, w2 is the attitude weight, σ is the balance weight, and E is the comprehensive loss error; c. Calculating the gradient of the loss function with respect to the parameters in the network through backpropagation; d. Calculating the new parameter values through the Adam optimizer according to its update rule using the gradient and the current parameter values, and updating the parameter values to move the parameters of the network in the direction of reducing the loss function; e. Repeating the above steps until the training of all batches of data is completed.

Images