Object identification method and device based on three-dimensional reconstruction
By combining multi-view geometry and NeRF, the problem of low computing efficiency in the prior art is solved, especially in the reconstruction of textureless areas, and more efficient and accurate object recognition effect is achieved.
Patent Information
- Application Number
- CN202411525920.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art has problems with low computing efficiency in three-dimensional reconstruction and object recognition, especially when dealing with textureless areas, it is difficult to effectively reconstruct depth and geometric information.
A three-dimensional reconstruction method combining multi-view geometry and NeRF is adopted to obtain the initial point cloud through multi-view geometry, and the NeRF neural network is used to fit colors and geometric features in texture-free areas to generate continuous three-dimensional representations. Then the point clouds obtained from NeRF and multi-view geometry are fused, corrected and integrated, and finally object recognition is used using a deep learning classification model.
It improves the efficiency and accuracy of object recognition, especially in three-dimensional reconstruction of textureless areas, which can more effectively reconstruct depth and geometric information, reducing production costs.
Smart Images
Figure CN119478919A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer image processing, and in particular to a method and device for object recognition based on three-dimensional reconstruction. Background Art
[0002] In the field of 3D reconstruction, the current mainstream solutions include methods based on multi-view geometry and NeRF.
[0003] The method based on multi-view geometry is a traditional method before the popularity of neural networks, with Colmap software as a typical representative. Colmap extracts geometric information from two-dimensional images from multiple perspectives and then derives the three-dimensional structure. The main processes include: feature extraction and matching, sparse reconstruction based on SfM, dense reconstruction based on MVS, and camera calibration and optimization based on BA. After the above process, the final three-dimensional model is obtained.
[0004] NeRF (Neural Radiance Fields) is a neural network-based 3D reconstruction and rendering technology that can generate high-quality 3D scene representations from a set of 2D images. The core idea is to learn the color and density distribution of each spatial position and viewpoint in the scene through a multi-layer perceptron (MLP) network to generate volume rendering output. However, multi-view-based solutions cannot handle areas without textures.
[0005] NeRF training requires input of images and camera parameters. The algorithm encodes each 3D coordinate with high frequency and inputs it into the neural network, which then outputs the color (RGB) and density (transparency of volume rendering) of the point. By comparing the generated image with the input real image, the network parameters are adjusted to make the generated image more closely match the real image. However, the NeRF-based solution has the problem of low computational efficiency.
[0006] In the field of point cloud-based 3D scene recognition, PointNet uses a shared multi-layer perceptron to learn the features of each point, which can directly process unordered point clouds. Based on this, scholars later proposed the PointNet++
[19] network, which adds the ability to extract and aggregate multi-level neighborhood features, and can better capture the local and global structures of point cloud data. DGCNN constructs the adjacency relationship of point cloud data through dynamic graphs, which can better capture the topological structure between points. The reconstructed model has irregular topology and is difficult to render again. Summary of the invention
[0007] The present invention aims to provide an object recognition method and device based on three-dimensional reconstruction to solve the deficiencies in the prior art. The technical problem to be solved by the present invention is achieved through the following technical solutions.
[0008] A first aspect of the present invention provides an object recognition method based on three-dimensional reconstruction, the method comprising: The 3D reconstruction step based on multi-view geometry uses a 3D reconstruction algorithm based on multi-view geometry, with multiple scene images as input and a point cloud of the scene as output; The texture-free region detection step uses the depth information obtained in the 3D reconstruction process of multi-view geometry to set a threshold. For the depth information of each photo, if the depth information is less than the threshold and there is no point cloud in the area, it is determined to be a texture-free area. The NeRF-based 3D reconstruction step processes the input image based on NeRF to generate the corresponding point cloud, which is implemented through the following sub-steps: Image input sub-step: NeRF relies on multi-view image input, which comes from multiple camera angles and covers different perspectives of the target scene to ensure the integrity and accuracy of the reconstruction; for textureless areas, NeRF's powerful neural network can be used to fit color and geometric features; Neural network modeling sub-step: NeRF uses an MLP multi-layer perceptron neural network to fit the radiation field of the scene. The neural network accepts each camera pose and light direction as input and outputs color and density values for each 3D spatial position, allowing NeRF to generate a continuous 3D representation and reconstruct depth and geometry even in textureless areas. Volume rendering sub-step: NeRF uses volume rendering technology to render the density and color values of the three-dimensional space back into a two-dimensional image, and gradually optimizes its loss function to generate a high-quality rendered image that matches the input photo; Point cloud extraction sub-step: After completing the reconstruction of the 3D scene, points with higher density can be extracted from the NeRF model as point cloud data, where the points represent the boundaries of objects or other important geometric features in the scene; Point cloud conversion sub-step: The scene output by NeRF can be converted into a 3D point cloud, where each point is represented by its position and color value. For textureless areas, NeRF will fit its geometric features based on the spatial information of the input image to generate the corresponding point cloud. Point cloud fusion step: align, correct and integrate the point clouds obtained from NeRF and multi-view geometry to ensure that each point cloud is in the same coordinate system; Point cloud classification step: Based on the deep learning classification model, objects are identified from the point cloud; each classified point cloud corresponds to a specific object category in the asset library.
[0009] In combination with the first aspect, the point cloud fusion step further includes: Obtaining the camera pose sub-step: In multi-view geometric reconstruction, each photo corresponds to a camera pose, including the camera's rotation and translation information; the pose is used to determine the spatial position of each picture in the scene; NeRF generated point cloud sub-step: The point cloud extracted from NeRF represents a dense and optimized spatial point distribution in the scene, including color and depth information; Point cloud sub-step of multi-view geometry: generate a set of preliminary point clouds based on different angles through SFM to recover structure from motion or MVS multi-view stereo matching algorithm; Fusion sub-step: Optimize by iterating the closest point ICP algorithm so that the two sets of point clouds are accurately matched in three-dimensional space.
[0010] In combination with the first aspect, the point cloud classification step further includes: Point cloud preprocessing sub-step: preprocessing the fused point cloud, the preprocessing is denoising, downsampling and point cloud normalization, to reduce the computational burden and improve the classification accuracy; Feature extraction sub-step: Point cloud classification relies on feature extraction algorithms to extract the geometric information and spatial distribution of each point into a high-dimensional feature vector. The feature extraction algorithms include local curvature, normal vector estimation, and point cloud convolutional neural networks based on deep learning. Classification model training sub-step: The deep learning-based classification models PointNet and PointCNN process sparse and irregular point cloud data and classify each point cloud based on global and local features; Object recognition and segmentation sub-step: classifying the objects to be recognized from the point cloud; the classified point cloud corresponds to a specific object category in the asset library.
[0011] In combination with the first aspect, the above point cloud classification step further includes a point cloud homogenization step: Mesh conversion sub-step: Use Marching Cubes or Poisson Surface Reconstruction mesh generation algorithm to convert point cloud data into triangular mesh representation, and use the triangular mesh to represent the connection mode of vertices to more accurately describe the surface structure of the object; Fixed resolution sampling sub-step: After generating the grid representation, resample the grid based on the set resolution; generate new point cloud data through a uniform sampling algorithm so that the data points of the new point cloud are more evenly distributed in space; Point cloud homogenization sub-step: The intervals between each point in the point cloud data are roughly equal, and can be evenly covered in different areas of the object to avoid the problem of excessive point density.
[0012] In combination with the first aspect, the point cloud classification step further includes an object classification step based on PCA: PCA feature extraction sub-step: perform principal component analysis PCA on the homogenized point cloud data, calculate the covariance matrix and extract the first few principal components, and select the first 10 principal components to extract. The principal components can describe most of the geometric features of the object; Align principal components: Align the extracted principal components with the principal components of similar objects in the asset library. Aligning the first principal component ensures that the two sets of point clouds are compared in the same direction. The alignment process involves rigid body transformation, which makes the objects geometrically aligned for subsequent classification. Calculate the principal component distance ion step: After completing the alignment, calculate the principal component distance between the current object and each similar object in the asset library. The principal component distance is based on the Euclidean distance measurement method to measure the similarity of two objects in a low-dimensional feature space. Objects with smaller principal component distances are considered to have higher similarity with the current object; Return the most similar object sub-step: According to the calculated principal component distance, select the object closest to the asset library, and the object is the classification result most similar to the current object.
[0013] A second aspect of the present invention provides an object recognition device based on three-dimensional reconstruction, the device comprising: The 3D reconstruction module based on multi-view geometry uses a 3D reconstruction algorithm based on multi-view geometry. The input is multiple scene images and the output is a point cloud of the scene. The texture-free area detection module uses the depth information obtained in the 3D reconstruction process of multi-view geometry to set a threshold. For the depth information of each photo, if the depth information is less than the threshold and there is no point cloud in the area, it is determined to be a texture-free area. The NeRF-based 3D reconstruction module processes the input image based on NeRF and generates the corresponding point cloud. This is achieved through the following sub-steps: Image input submodule: NeRF relies on multi-view image input, which comes from multiple camera angles and covers different perspectives of the target scene to ensure the integrity and accuracy of reconstruction; for textureless areas, NeRF's powerful neural network can be used to fit color and geometric features; Neural network modeling submodule: NeRF uses an MLP multi-layer perceptron neural network to fit the radiation field of the scene. The neural network accepts each camera pose and light direction as input and outputs color and density values for each 3D spatial position, allowing NeRF to generate a continuous 3D representation and reconstruct depth and geometry even in textureless areas. Volume rendering submodule: NeRF uses volume rendering technology to render the density and color values of the three-dimensional space back into a two-dimensional image, and gradually optimizes its loss function to generate a high-quality rendered image that matches the input photo; Point cloud extraction submodule: After completing the reconstruction of the 3D scene, high-density points can be extracted from the NeRF model as point cloud data, which represent the boundaries of objects or other important geometric features in the scene; Point cloud conversion submodule: The scene output by NeRF can be converted into a 3D point cloud, where each point is represented by its position and color value. For textureless areas, NeRF will fit its geometric features based on the spatial information of the input image to generate the corresponding point cloud. Point cloud fusion module: aligns, corrects and integrates the point clouds obtained from NeRF and multi-view geometry to ensure that each point cloud is in the same coordinate system; Point cloud classification module: Based on the deep learning classification model, objects are identified from the point cloud; each classified point cloud corresponds to a specific object category in the asset library.
[0014] The present invention integrates multi-view and NeRF solutions for three-dimensional reconstruction, which can be applied in film and television production, virtual reality and augmented reality to improve the recognition efficiency and accuracy of objects and reduce production costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a flowchart of the steps of an object recognition method based on three-dimensional reconstruction of the present invention; Figure 2 It is an internal structure diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0016] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0017] Abbreviations and key terms commonly used in this field are defined as follows: Point cloud refers to a set of discrete 3D data points collected by 3D scanning or stereo vision, each of which contains spatial coordinates (such as x, y, z). Point cloud is usually used to represent the surface structure of objects or scenes, and is widely used in computer vision, 3D reconstruction, autonomous driving, virtual reality and other fields.
[0018] A mesh is a three-dimensional geometric structure consisting of vertices, edges, and faces, and is usually used to represent the surface of an object. A mesh model can accurately describe the shape of an object and is particularly suitable for scene and object representation in 3D modeling, animation, game development, and computer graphics.
[0019] NeRF is a 3D scene representation method based on neural networks. By mapping any point in 3D space to color and volume density, it can generate high-quality 3D reconstruction from 2D images. NeRF is widely used in new perspective synthesis, 3D reconstruction and virtual reality.
[0020] Multi-view geometry is a field that studies the geometric relationships of 3D structures derived from images acquired from multiple viewpoints. By analyzing multiple images of the same scene taken by cameras at different positions, multi-view geometry technology can reconstruct the 3D shape of the scene and is widely used in 3D reconstruction and pose estimation in computer vision.
[0021] Classification is a task in machine learning that aims to assign input data to predefined categories or labels. Classification algorithms usually train models to recognize patterns or features in input data and classify new data based on these features. Classification is widely used in fields such as image recognition, text analysis, and speech recognition.
[0022] PCA (Principal Component Analysis) is a statistical method for dimensionality reduction that converts high-dimensional data into a low-dimensional space while retaining the main features or variance of the data. PCA is widely used in scenarios such as data compression, feature extraction, and noise filtering, especially when dealing with high-dimensional data sets.
[0023] like Figure 1 As shown, the specific implementation process of the present invention is as follows: 1) 3D reconstruction based on multi-view geometry. You can use any of the current mainstream 3D reconstruction algorithms based on multi-view geometry, with multiple scene images as input and point clouds of the scene as output.
[0024] 2) Textureless area detection. The depth information obtained in the 3D reconstruction process of multi-view geometry is used to set a threshold. For the depth information of each photo, if the depth value is less than the threshold and there is no point cloud in the area, it is determined to be a textureless area.
[0025] 3) 3D reconstruction based on NeRF.
[0026] a) Input photos: NeRF relies on multi-view photo input. These photos can come from multiple camera angles, covering different perspectives of the target scene, and are preferably evenly distributed to ensure the completeness and accuracy of the reconstruction. Especially for textureless areas, NeRF's powerful neural network can be used to fit color and geometric features.
[0027] b) Neural Network Modeling: NeRF uses an MLP (Multi-layer Perceptron) neural network to fit the radiation field of the scene. The network accepts each camera pose and light direction as input and outputs color and density values for each 3D spatial position. This allows NeRF to generate a continuous 3D representation, able to reconstruct depth and geometry even in textureless areas.
[0028] c) Volume Rendering: NeRF uses volume rendering technology to render the density and color values of the three-dimensional space back into a two-dimensional image, and gradually optimizes its loss function to generate high-quality rendered images that match the input photo.
[0029] d) Point cloud extraction: After completing the reconstruction of the 3D scene, high-density points can be extracted from the NeRF model as point cloud data. These points usually represent the boundaries of objects or other important geometric features in the scene.
[0030] e) Conversion to point cloud: The scene output by NeRF can be converted into a 3D point cloud, where each point is represented by its position and color value. For textureless areas, NeRF will fit its geometric features based on the spatial information of the input image to generate the corresponding point cloud.
[0031] 4) Point cloud fusion.
[0032] a) Obtaining camera pose: In multi-view geometric reconstruction, each photo corresponds to a camera pose (including camera rotation and translation information). These poses are used to determine the spatial position of each image in the scene.
[0033] b) Point cloud generated by NeRF: The point cloud extracted from NeRF represents a denser and optimized spatial point distribution in the scene, including color and depth information.
[0034] c) Point clouds from multi-view geometry: Through traditional multi-view geometry algorithms, such as SFM (Structure from Motion) or MVS (Multi-view Stereo Matching), a set of preliminary point clouds based on different angles can also be generated. These point clouds are usually sparse, but can provide additional geometric constraints for the scene.
[0035] d) Fusion process: The point cloud fusion process involves aligning, correcting, and integrating the point clouds obtained from different methods (NeRF and multi-view geometry). By aligning the camera pose information, it is possible to ensure that each point cloud is in the same coordinate system. Further fusion can be optimized by techniques such as the iterative closest point algorithm (ICP) so that the two sets of point clouds are accurately matched in three-dimensional space.
[0036] 5) Point cloud classification.
[0037] a) Point cloud preprocessing: First, the fused point cloud is preprocessed, such as denoising, downsampling, and point cloud normalization, to reduce the computational burden and improve the classification accuracy.
[0038] b) Feature extraction: Point cloud classification usually relies on feature extraction algorithms to extract the geometric information and spatial distribution of each point into a high-dimensional feature vector. Common feature extraction methods include local curvature, normal vector estimation, and point cloud convolutional neural networks based on deep learning (such as PointNet).
[0039] c) Classification model training: Classification models based on deep learning, especially PointNet and PointCNN, can process sparse and irregular point cloud data and classify each point based on global and local features.
[0040] d) Object recognition and segmentation: After classification, the identified objects are segmented from the point cloud. Each classified point corresponds to a specific object category in the asset library.
[0041] 6) Point cloud homogenization.
[0042] a) Point cloud to mesh conversion: Use mesh generation algorithms (such as Marching Cubes or Poisson Surface Reconstruction) to convert point cloud data into triangular mesh representation. This conversion can more accurately describe the surface structure of the object through the connection of mesh vertices.
[0043] b) Fixed resolution sampling: After generating the grid representation, the grid is resampled based on the set resolution. New point cloud data is generated through a uniform sampling algorithm, making the data points of the new point cloud more evenly distributed in space. This is critical for the subsequent analysis and comparison process.
[0044] c) Point cloud homogenization result: The final point cloud data has roughly equal intervals between each point, and can evenly cover different areas of the object, avoiding the problem of excessive point density.
[0045] 7) Object classification based on PCA.
[0046] a) PCA feature extraction: Perform principal component analysis (PCA) on the homogenized point cloud data. The goal of PCA is to convert high-dimensional point cloud data into a low-dimensional feature space, usually by calculating the covariance matrix and extracting the first few principal components. You can choose to extract the first 10 principal components, which usually describe most of the geometric features of the object.
[0047] b) Align principal components: Align the extracted principal components with the principal components of similar objects in the asset library. Usually, the first principal component is aligned to ensure that the two sets of point clouds are compared in the same direction. The alignment process may involve rigid body transformation, so that the objects are geometrically aligned to facilitate subsequent classification.
[0048] c) Calculate the distance between principal components: After alignment is completed, calculate the distance between the principal components of the current object and each similar object in the asset library. This distance can be used to measure the similarity of two objects in a low-dimensional feature space based on metrics such as Euclidean distance. Objects with smaller principal component distances are considered to have higher similarity with the current object.
[0049] d) Return the most similar object: Based on the calculated principal component distance, select the object in the asset library that is closest to the current object. This object is the classification result that is most similar to the current object and can usually be used for further object recognition or classification tasks.
[0050] Figure 2 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Figure 2 As shown. The electronic device includes a processor, an internal memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection.
[0051] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0052] The present invention is planned to be applied to China Unicom cloud disk products. It provides innovative solutions for multiple industries by extracting key information and intelligently classifying it. In the fields of film and television production, virtual reality (VR) and augmented reality (AR), this technology can significantly improve creative efficiency and reduce production costs. For example, complex backgrounds or props can be quickly constructed in film and television special effects; realistic environment and object models can be automatically generated during game design; and a richer and more personalized user experience can be provided in VR / AR applications. With the rapid growth of the digital content industry, the demand for efficient generation of high-quality visual content is increasing. Therefore, the present invention not only meets the existing market demand, but also has the potential to open up new application scenarios, such as the production of interactive teaching materials in online education.
[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application.
[0054] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.
[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for object recognition based on three-dimensional reconstruction, characterized in that: The method comprises: The 3D reconstruction step based on multi-view geometry uses a 3D reconstruction algorithm based on multi-view geometry, with multiple scene images as input and a point cloud of the scene as output; The texture-free area detection step uses the depth information obtained in the 3D reconstruction process of multi-view geometry to set a threshold. For the depth information of each photo, if the depth information is less than the set threshold and there is no point cloud in the area, it is determined to be a texture-free area. The NeRF-based 3D reconstruction step processes the input image based on NeRF to generate the corresponding point cloud, which is implemented through the following sub-steps: Image input sub-step: NeRF relies on multi-view image input, which comes from multiple camera angles and covers different perspectives of the target scene to ensure the integrity and accuracy of the reconstruction; for textureless areas, NeRF's powerful neural network is used to fit color and geometric features; Neural network modeling sub-step: NeRF uses an MLP multi-layer perceptron neural network to fit the radiation field of the scene. The neural network accepts each camera pose and light direction as input and outputs color and density values for each 3D spatial position, allowing NeRF to generate a continuous 3D representation and reconstruct depth and geometry even in textureless areas. Volume rendering sub-step: NeRF uses volume rendering technology to render the density and color values of the three-dimensional space back into a two-dimensional image, and gradually optimizes its loss function to generate a high-quality rendered image that matches the input photo; Point cloud extraction sub-step: After completing the reconstruction of the 3D scene, points with higher density can be extracted from the NeRF model as point cloud data, where the points represent the boundaries of objects or other important geometric features in the scene; Point cloud conversion sub-step: The scene output by NeRF can be converted into a 3D point cloud, where each point is represented by its position and color value. For textureless areas, NeRF will fit its geometric features based on the spatial information of the input image to generate the corresponding point cloud. Point cloud fusion step: align, correct and integrate the point clouds obtained from NeRF and multi-view geometry to ensure that each point cloud is in the same coordinate system; Point cloud classification step: Based on the deep learning classification model, objects are identified from the point cloud; each classified point cloud corresponds to a specific object category in the asset library.
2. The object recognition method based on three-dimensional reconstruction according to claim 1, characterized in that: The point cloud fusion step further comprises: Obtaining the camera pose sub-step: In multi-view geometric reconstruction, each photo corresponds to a camera pose, including the camera's rotation and translation information; the pose is used to determine the spatial position of each picture in the scene; NeRF generated point cloud sub-step: The point cloud extracted from NeRF represents a dense and optimized spatial point distribution in the scene, including color and depth information; Point cloud sub-step of multi-view geometry: generate a set of preliminary point clouds based on different angles through SFM to recover structure from motion or MVS multi-view stereo matching algorithm; Fusion sub-step: Optimize by iterating the closest point ICP algorithm so that the two sets of point clouds are accurately matched in three-dimensional space.
3. The object recognition method based on three-dimensional reconstruction according to claim 1, characterized in that: The point cloud classification step further comprises: Point cloud preprocessing sub-step: preprocessing the fused point cloud, the preprocessing is denoising, downsampling and point cloud normalization, to reduce the computational burden and improve the classification accuracy; Feature extraction sub-step: Point cloud classification relies on feature extraction algorithms to extract the geometric information and spatial distribution of each point cloud into a high-dimensional feature vector. The feature extraction algorithms include local curvature, normal vector estimation, and point cloud convolutional neural networks based on deep learning. Classification model training sub-step: The deep learning-based classification models PointNet and PointCNN process sparse and irregular point cloud data and classify each point cloud based on global and local features; Object recognition and segmentation sub-step: classifying the objects to be recognized from the point cloud; the classified point cloud corresponds to a specific object category in the asset library.
4. The object recognition method based on three-dimensional reconstruction as claimed in claim 3, characterized in that: The point cloud classification step further includes a point cloud homogenization step, which is specifically divided into the following sub-steps: Mesh conversion sub-step: Use Marching Cubes or Poisson Surface Reconstruction mesh generation algorithm to convert point cloud data into triangular mesh representation, and use the triangular mesh to represent the connection mode of vertices to more accurately describe the surface structure of the object; Fixed resolution sampling sub-step: after generating the grid representation, re-sampling the grid based on the set resolution; generating new point cloud data through a uniform sampling algorithm, so that the new point cloud data is more evenly distributed in space; Point cloud homogenization sub-step: The intervals between each point in the point cloud data are roughly equal, and can be evenly covered in different areas of the object, avoiding the problem of excessive concentration of point cloud data density.
5. The object recognition method based on three-dimensional reconstruction according to claim 4, characterized in that: The point cloud classification step further includes an object classification step based on PCA: PCA feature extraction sub-step: perform principal component analysis PCA on the homogenized point cloud data, calculate the covariance matrix and extract the first few principal components, and select the first 10 principal components to extract. The principal components can describe most of the geometric features of the object; Align principal components: Align the extracted principal components with the principal components of similar objects in the asset library. Aligning the first principal component ensures that the two sets of point clouds are compared in the same direction. The alignment process involves rigid body transformation, which makes the objects geometrically aligned for subsequent classification. Calculate the principal component distance ion step: After the alignment is completed, calculate the principal component distance between the current object and each similar object in the asset library. The principal component distance is based on the Euclidean distance measurement method to measure the similarity of two objects in a low-dimensional feature space. Objects with smaller principal component distances are considered to have higher similarity with the current object; Return the most similar object sub-step: According to the calculated distance between principal components, select the object with the closest distance in the asset library, and the object is the classification result most similar to the current object.
6. An object recognition device based on three-dimensional reconstruction, characterized in that: The device comprises: The 3D reconstruction module based on multi-view geometry uses a 3D reconstruction algorithm based on multi-view geometry. The input is multiple scene images and the output is a point cloud of the scene. The texture-free area detection module uses the depth information obtained in the 3D reconstruction process of multi-view geometry to set a threshold. For the depth information of each photo, if the depth information is less than the threshold and there is no point cloud in the area, it is determined to be a texture-free area. The NeRF-based 3D reconstruction module processes the input image based on NeRF and generates the corresponding point cloud. This is achieved through the following sub-steps: Image input submodule: NeRF relies on multi-view image input, which comes from multiple camera angles and covers different perspectives of the target scene to ensure the integrity and accuracy of reconstruction; for textureless areas, NeRF's powerful neural network can be used to fit color and geometric features; Neural network modeling submodule: NeRF uses an MLP multi-layer perceptron neural network to fit the radiation field of the scene. The neural network accepts each camera pose and light direction as input and outputs color and density values for each 3D spatial position, allowing NeRF to generate a continuous 3D representation and reconstruct depth and geometry even in textureless areas. Volume rendering submodule: NeRF uses volume rendering technology to render the density and color values of the three-dimensional space back into a two-dimensional image, and gradually optimizes its loss function to generate a high-quality rendered image that matches the input photo; Point cloud extraction submodule: After completing the reconstruction of the 3D scene, high-density points can be extracted from the NeRF model as point cloud data, which represent the boundaries of objects or other important geometric features in the scene; Point cloud conversion submodule: The scene output by NeRF can be converted into a 3D point cloud, where each point is represented by its position and color value. For textureless areas, NeRF will fit its geometric features based on the spatial information of the input image to generate the corresponding point cloud. Point cloud fusion module: aligns, corrects and integrates the point clouds obtained from NeRF and multi-view geometry to ensure that each point cloud is in the same coordinate system; Point cloud classification module: Based on the deep learning classification model, objects are identified from the point cloud; each classified point cloud corresponds to a specific object category in the asset library.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the object recognition method based on three-dimensional reconstruction described in any one of claims 1 to 5 are implemented.
8. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the object recognition method based on three-dimensional reconstruction as described in any one of claims 1 to 5.