Target object refined three-dimensional reconstruction method based on YOLOv8 labeling

Abstract

The invention discloses a target object refined three-dimensional reconstruction method based on YOLOv8 labeling, and the method comprises the following steps: constructing a YOLOv8 labeling model, carrying out the boundary labeling of a target object through the YOLOv8 labeling model, and recognizing an unshielded region and a target region; constructing a first HashNeRF model, inputting image data of a plurality of visual angles, performing global three-dimensional reconstruction of the target object through the first HashNeRF model, and outputting a color map and a depth map of each visual angle; based on the two-dimensional coordinates of the target area identified by the YOLOv8 labeling model and the depth map output by the first HashNeRF model, coordinate conversion of the target area is carried out; and constructing a second HashNeRF model, carrying out refined reconstruction on the target area under different visual angles through the second HashNeRF model, and identifying the occlusion area according to the depth maps of different visual angles to complete high-precision restoration of the target area and three-dimensional reconstruction of the target object. According to the invention, the technical problems of insufficient shielding region reconstruction precision and poor multi-view target region consistency in three-dimensional scene reconstruction through the NeRF technology in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for fine-grained three-dimensional reconstruction of a target object based on YOLOv8 annotation. Background Art

[0002] Neural Radiance Fields (NeRF) is a 3D scene reconstruction technology based on neural networks. It can generate highly realistic 3D scenes by learning the color and collective features of the scene from multiple perspective images. NeRF uses ray tracing and volume rendering technology to integrate images from different perspectives into a unified radiation field to achieve a realistic reproduction of the scene. However, there is a key challenge in the application of NeRF, that is, when the target object is occluded in certain perspectives, the model may not be able to fully reconstruct the structure and details of the area. In addition, NeRF technology focuses on the overall reconstruction of the scene, and its model usually learns and reconstructs from a global perspective, thereby ignoring the fine processing of specific target areas in the scene. The existing NeRF technology relies on multi-view images for 3D scene reconstruction. However, when some viewpoints in the input image are occluded, NeRF cannot learn the complete target object information from these viewpoints, resulting in incomplete reconstruction of some areas. In addition, NeRF cannot effectively distinguish the target object from the background, resulting in wasted computing resources on irrelevant background areas, affecting the modeling quality of the target area, that is, background interference exists. In recent years, some technologies have attempted to predict the content of the occluded area through image inference or completion technology of neural networks, which usually rely on the pixel information near the occluded area. Although the occluded area can be repaired to a certain extent, it is impossible to use multi-view information for unified reconstruction of the 3D scene. Or they try to deal with the occlusion problem in a single image, using semantic segmentation or deep learning to predict and repair the target area. The effect of this method usually depends on the context information of a single image, and it is difficult to achieve consistent completion across viewpoints. It can be seen that the above technical solutions cannot fully utilize the advantages of multi-view images and efficiently handle the situation where the target area is occluded in complex scenes. Therefore, it is urgent to propose a refined 3D reconstruction method for target objects based on YOLOv8 annotation to solve the technical problems of insufficient reconstruction accuracy of occluded areas and poor consistency of multi-view target areas in the existing 3D scene reconstruction using NeRF technology. Summary of the invention

[0003] The main purpose of the invention is to propose a refined 3D reconstruction method for target objects based on YOLOv8 annotation, aiming to solve the technical problems of insufficient reconstruction accuracy of occluded areas and poor consistency of multi-view target areas in the existing 3D scene reconstruction using NeRF technology.

[0004] To achieve the above object, the present invention provides a method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation, wherein the method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation comprises the following steps:

[0005] S1. Build a YOLOv8 annotation model, and use the YOLOv8 annotation model to annotate the boundaries of the target object and identify the unobstructed area and the target area;

[0006] S2. Build a first HashNeRF model, input image data from multiple perspectives, perform global 3D reconstruction of the target object through the first HashNeRF model, and output a color map and a depth map for each perspective;

[0007] S3, based on the two-dimensional coordinates of the target area identified by the YOLOv8 annotation model and the depth map output by the first HashNeRF model, performing coordinate transformation of the target area;

[0008] S4. Construct a second HashNeRF model, use the second HashNeRF model to perform fine reconstruction of the target area after coordinate transformation under different perspectives, and identify the occluded area according to the depth map of different perspectives, so as to complete the high-precision restoration of the target area and the three-dimensional reconstruction of the target object.

[0009] In one of the preferred solutions, step S1 is specifically:

[0010] The YOLOv8 annotation model is used to detect each frame of the input image data, generate a mask of the target area, annotate the target area in the image, and output the two-dimensional coordinates and bounding box size of the target area.

[0011] In one of the preferred solutions, step S2 is specifically:

[0012] S21, inputting the three-dimensional space coordinates and viewing directions of multiple viewing angle images, and using hash coding to map the three-dimensional space coordinates to a high-dimensional feature space;

[0013] S22, performing three-dimensional linear interpolation and splicing on the encoding results of different resolutions according to the characteristics of the hash table to form a feature vector containing global and local information of the scene;

[0014] S23, based on the concatenated feature vector, extracting the density and color information of the scene from the high-dimensional encoding space through a multi-layer perceptron, and calculating the final color value of each ray in the scene according to the density and color information of the scene;

[0015] S24. Construct a loss function, optimize the network weights of the first HashNeRF model through back propagation, realize global 3D reconstruction of the target object, and output color maps and depth maps of each perspective.

[0016] In one of the preferred solutions, step S3 is specifically:

[0017] According to the depth map of each viewing angle output by the first HashNeRF model, the depth value at the current viewing angle is obtained;

[0018] Calculate the 3D points of the target area based on the 2D coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle;

[0019] Convert the 3D points in the target area into 3D points in the world coordinate system according to the current camera position;

[0020] The three-dimensional point in the world coordinate system is converted into the three-dimensional coordinate in the camera coordinate system of the new perspective, and the three-dimensional coordinate in the camera coordinate system is projected onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target area in the new perspective.

[0021] One of the preferred solutions is to calculate the three-dimensional points of the target area according to the two-dimensional coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle, specifically:

[0022] Get the camera intrinsic parameter matrix of the current viewing angle; the camera intrinsic parameter matrix is:

[0023]

[0024] Among them, K is the camera internal parameter matrix, f x and f y is the focal length of the camera, c x and c y is the coordinate of the center point of the image;

[0025] According to the two-dimensional coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle, the camera intrinsic parameter matrix is ​​used to convert the two-dimensional coordinates of the target area into three-dimensional points in the camera coordinate system, specifically:

[0026]

[0027] Among them, (u, v) is the two-dimensional coordinate of the target area under the current viewing angle, Z c is the depth value at the corresponding viewing angle, [X c ,Y c ,Z c ] T is the 3D point coordinate in the camera coordinate system.

[0028] One of the preferred solutions is that the three-dimensional points of the target area are converted into three-dimensional points in the world coordinate system according to the current camera position, specifically:

[0029] Obtain the camera extrinsic parameters of the current viewing angle; the camera extrinsic parameters include the rotation matrix and displacement vector of the camera;

[0030] According to the camera external parameters, the three-dimensional points in the target area are converted into three-dimensional points in the world coordinate system, specifically:

[0031]

[0032] Among them, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system, R A is the rotation matrix at the current viewing angle, t A is the displacement vector at the current viewing angle, [X c ,Y c ,Z c ] T is the 3D point coordinate in the camera coordinate system.

[0033] One of the preferred solutions is that the three-dimensional point in the world coordinate system is converted into a three-dimensional coordinate in the camera coordinate system of the new perspective, specifically:

[0034] Get the three-dimensional point of the target area in the world coordinate system, and convert it into the three-dimensional coordinates of the new perspective in the camera coordinate system according to the camera external parameters of the new perspective; the three-dimensional coordinates of the target area in the camera coordinate system under the new perspective are:

[0035]

[0036] Among them, [X c' ,Y c' ,Z c' ] T is the 3D coordinate of the target area in the camera coordinate system under the new perspective, R B is the rotation matrix under the new perspective, t B is the displacement vector at the new viewing angle, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system.

[0037] One of the preferred solutions is to project the three-dimensional coordinates in the camera coordinate system onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area in the new viewing angle, specifically:

[0038] According to the camera intrinsic parameter matrix of the new viewing angle, the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area at the new viewing angle. The two-dimensional coordinates of the target area at the new viewing angle are:

[0039]

[0040] Among them, (u', v') is the two-dimensional coordinate of the target area at the new perspective, [X c' ,Y c' ,Z c' ] T is the 3D coordinate of the target area in the camera coordinate system under the new perspective, K B is the camera intrinsic parameter matrix under the new perspective.

[0041] One of the preferred solutions, step S4 uses the second HashNeRF model to perform fine reconstruction of the target area after coordinate transformation under different viewing angles, specifically:

[0042] Input the perspective pose under the new perspective and the two-dimensional coordinates of the target area under the new perspective, and generate the color map and depth map of the position coordinates under the new perspective through the second HashNeRF model.

[0043] In one preferred solution, the step S4 identifies the occluded area according to the depth maps of different viewing angles, specifically:

[0044] When switching to a different viewing angle, a global view of the target object under the viewing angle is generated through the first HashNeRF model, and the depth information under the switched viewing angle is obtained;

[0045] The three-dimensional coordinates of the target area annotated by the YOLOv8 annotation model are mapped to the switching perspective through coordinate transformation to obtain the target area under the switching perspective;

[0046] The second HashNeRF model is used to achieve fine reconstruction of the target area under the switching perspective, that is, to perform high-precision identification and restoration of the occluded area and render high-precision details of the target area.

[0047] In the above technical scheme of the present invention, the method for fine three-dimensional reconstruction of the target object based on YOLOv8 annotation includes the following steps: constructing a YOLOv8 annotation model, and annotating the boundary of the target object through the YOLOv8 annotation model, identifying the unobstructed area and the target area; constructing a first HashNeRF model, inputting image data of multiple perspectives, performing global three-dimensional reconstruction of the target object through the first HashNeRF model, and outputting the color map and depth map of each perspective; performing coordinate conversion of the target area based on the two-dimensional coordinates of the target area identified by the YOLOv8 annotation model and the depth map output by the first HashNeRF model; constructing a second HashNeRF model, performing fine reconstruction of the target area after coordinate conversion under different perspectives through the second HashNeRF model, and identifying the occluded area according to the depth map of different perspectives, thereby completing the high-precision restoration of the target area and the three-dimensional reconstruction of the target object. The present invention solves the technical problems of insufficient reconstruction accuracy of the occluded area and poor consistency of the multi-perspective target area in the existing three-dimensional scene reconstruction by NeRF technology.

[0048] In the present invention, by setting a first HashNeRF model and a second HashNeRF model, the global scene is reconstructed through the first HashNeRF model. When rendering the scene, the second HashNeRF model uses the depth information provided by the first HashNeRF model to finely reconstruct the target area at a specific perspective, which not only ensures the integrity of the global scene, but also achieves high-precision detail presentation in the local area.

[0049] In the present invention, the depth map generated by the first HashNeRF model is combined with the target area annotated by the YOLOv8 annotation model to convert and compare between different perspectives, and the coordinates of the target area are directly mapped to different perspectives. There is no need to rely on the global model to speculate and infer the occluded area, thereby improving the reconstruction accuracy of the occluded area and overcoming the shortcomings of the prior art that the occluded area is inaccurate due to reliance on the global model to infer the occluded area; and combined with the depth map and camera pose information, accurate coordinate conversion of the target area can be achieved, making the reconstruction results under different perspectives more coordinated and consistent.

[0050] In the present invention, the second HashNeRF model is used to focus on learning and reconstructing the target area, and combined with the target area labeled by the YOLOv8 labeling model, a stronger detail recovery capability is provided when the occluded area is re-observed, thereby avoiding the defect of insufficiently fine recovery of the occluded area and improving the detail recovery capability of the occluded area. At the same time, the reconstruction task is divided into global reconstruction and fine reconstruction of the target area, reducing unnecessary global detail processing burden, thereby improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of a method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation according to an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of the YOLOv8 annotation model according to an embodiment of the present invention;

[0054] Figure 3 Schematic diagram of the first HashNeRF model of an embodiment of the present invention.

[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with the implementation methods and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In addition, in the present invention, the descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0058] Furthermore, the technical solutions between the various embodiments of the present invention may be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in the field. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] See also Figure 1 According to one aspect of the present invention, the present invention provides a method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation, wherein the method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation comprises the following steps:

[0060] S1. Build a YOLOv8 annotation model, and use the YOLOv8 annotation model to annotate the boundaries of the target object and identify the unobstructed area and the target area;

[0061] S2. Build a first HashNeRF model, input image data from multiple perspectives, perform global 3D reconstruction of the target object through the first HashNeRF model, and output a color map and a depth map for each perspective;

[0062] S3, based on the two-dimensional coordinates of the target area identified by the YOLOv8 annotation model and the depth map output by the first HashNeRF model, performing coordinate transformation of the target area;

[0063] S4. Construct a second HashNeRF model, use the second HashNeRF model to perform fine reconstruction of the target area after coordinate transformation under different perspectives, and identify the occluded area according to the depth map of different perspectives, so as to complete the high-precision restoration of the target area and the three-dimensional reconstruction of the target object.

[0064] Specifically, in this embodiment, see Figure 2 , which is a schematic diagram of the YOLOv8 annotation model, through which each frame of the input image data is detected, a mask of the target area is generated, the target area in the image is annotated, and the two-dimensional coordinates and bounding box size of the target area are output; the target area of ​​the input multiple frames of the image is identified through the YOLOv8 annotation model, the two-dimensional coordinates, bounding box or mask of the target object are output, and the target object is annotated and tracked in multiple perspectives, providing basic data for subsequent refined reconstruction.

[0065] Specifically, in this embodiment, see Figure 3 , which is the network structure of the first HashNeRF model, and the global three-dimensional scene of the target object is reconstructed through the first HashNeRF model; the first HashNeRF model is an improved network based on neural radiation field, which is used for efficient three-dimensional scene reconstruction and rendering. By combining hash coding volume rendering, while maintaining high rendering quality, the training and calculation speeds are greatly improved.

[0066] Specifically, in this embodiment, the step S2 is specifically:

[0067] S21. Input the three-dimensional spatial coordinates and viewing directions of multiple view images, and use hash coding to map the three-dimensional spatial coordinates to a high-dimensional feature space; encode the input three-dimensional spatial coordinates through hash tables of different resolutions to form multi-scale spatial features. These encoders are similar to the position information encoding in traditional neural radiation fields, but through the sparse representation of hash tables, the memory usage and computational overhead are greatly reduced; each layer in the hash table corresponds to a different resolution, providing a multi-level detail capture capability from low resolution to high resolution, which helps model local details in complex scenes;

[0068] S22, using features of multiple hash tables, performing three-dimensional linear interpolation and splicing on encoding results of different resolutions to form a feature vector containing global and local information of the scene; the feature vector retains the global and local information of the scene and can capture complex geometric structures and texture changes;

[0069] S23. Based on the spliced ​​feature vector, the density and color information of the scene are extracted from the high-dimensional coding space through a multi-layer perceptron, and the final color value of each ray in the scene is calculated according to the density and color information of the scene; wherein, density prediction, a part of the multi-layer perceptron outputs volume density, and the density value is used to determine the position of obstacles encountered by the ray in the scene; color prediction, another part of the multi-layer perceptron calculates the color value of the voxel in combination with the viewing direction, so that the scene is more realistic at different viewing angles; wherein, the final color value of each ray in the scene is calculated using a volume rendering formula according to the density and color information of the scene, specifically: the ray passes through the three-dimensional scene, the density value is integrated along the direction of the ray, and the contribution of each voxel is accumulated; the color value is obtained by weighted summing the color value of each voxel, and the weight depends on the density of the voxel and the transmittance of the ray at the voxel;

[0070] S24. Construct a loss function, optimize the network weights of the first HashNeRF model through back propagation, realize global three-dimensional reconstruction of the target object, and output color maps and depth maps of each perspective; specifically, during the training process of the first HashNeRF model, use the difference between the synthesized 2D image and the actual 2D observed image as the loss function, optimize the network weights of the first HashNeRF model through back propagation, and through the combination of hash coding and neural radiation field, enable the model to effectively learn the three-dimensional structure of complex scenes with less computing resources and time.

[0071] Specifically, in this embodiment, the first HashNeRF model can perform query and difference operations. Compared with traditional models, the first HashNeRF model avoids unnecessary calculations through sparse representation and multi-resolution features. Therefore, it can ensure that the rendering speed is greatly accelerated while maintaining accuracy; at the same time, the first HashNeRF model efficiently represents the space through multi-resolution hash coding, and then uses a geometrically lightweight multi-layer perceptron to predict the density and color of the scene, and finally achieves high-quality image generation through rendering, that is, realizes global three-dimensional reconstruction of the target object, ensuring that the first HashNeRF model can achieve efficient rendering and high-fidelity scene reconstruction in complex scenes, reducing the complexity of calculation.

[0072] Specifically, in this embodiment, the first HashNeRF model and the second HashNeRF model have the same network structure. The trained first HashNeRF model and the second HashNeRF model obtain the color map and depth map of the position mark under the viewing angle by inputting the viewing angle information and the position coordinates under the viewing angle. The network structure of the first HashNeRF model and the second HashNeRF model is shown in Figure 3 .

[0073] Specifically, in this embodiment, the multi-view image is globally reconstructed through the first HashNeRF model to generate a color map and a depth map of the scene. The first HashNeRF model provides a global three-dimensional structure and depth information at each view as a geometric basis for fine reconstruction; and according to the two-dimensional coordinates annotated by the YOLOv8 annotation model, the depth map generated by the global three-dimensional reconstruction and the camera pose at the corresponding view, the two-dimensional coordinates of the target area are converted into three-dimensional coordinates at different viewpoints, and the position of the target area in multiple viewpoints is accurately located; through the multi-view depth map and camera pose information, the two-dimensional coordinates annotated by the YOLOv8 annotation model are converted into three-dimensional space coordinates, thereby realizing reconstruction at different viewpoints, thereby enhancing the compatibility and coordination between viewpoints.

[0074] Specifically, in this embodiment, step S3 is as follows:

[0075] According to the depth map of each viewing angle output by the first HashNeRF model, the depth value at the current viewing angle is obtained;

[0076] Calculate the 3D points of the target area based on the 2D coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle;

[0077] Convert the 3D points in the target area into 3D points in the world coordinate system according to the current camera position;

[0078] The three-dimensional point in the world coordinate system is converted into the three-dimensional coordinate in the camera coordinate system of the new perspective, and the three-dimensional coordinate in the camera coordinate system is projected onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target area in the new perspective.

[0079] Specifically, in this embodiment, the three-dimensional points of the target area are calculated according to the two-dimensional coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle, specifically:

[0080] Get the camera intrinsic parameter matrix of the current viewing angle; the camera intrinsic parameter matrix is:

[0081]

[0082] Among them, K is the camera internal parameter matrix, f x and f y is the focal length of the camera, c x and c y The center point of the image is

[0083] According to the two-dimensional coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle, the camera intrinsic parameter matrix is ​​used to convert the two-dimensional coordinates of the target area into three-dimensional points in the camera coordinate system, specifically:

[0084]

[0085] Among them, (u, v) is the two-dimensional coordinate of the target area under the current viewing angle, Z c is the depth value at the corresponding viewing angle, [X c ,Y c ,Z c ] T is the 3D point coordinate in the camera coordinate system.

[0086] Specifically, in this embodiment, the three-dimensional points of the target area are converted into three-dimensional points in the world coordinate system according to the current viewing angle camera posture, specifically:

[0087] Get the camera extrinsic parameters of the current viewing angle; the camera extrinsic parameters include the camera's rotation matrix R and displacement vector t; the camera extrinsic parameters describe the position of the camera in the world coordinate system, that is, the camera's rotation matrix and unique vector, which are used to convert the point of the world coordinate system to the camera coordinate system; specifically:

[0088] [X c ,Y c ,Z c ] T =R[X w ,Y w ,Z w ] T +t

[0089] According to the camera external parameters, the three-dimensional points in the target area are converted into three-dimensional points in the world coordinate system, specifically:

[0090]

[0091] Among them, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system, R A is the rotation matrix at the current viewing angle, t A is the displacement vector at the current viewing angle, [X c ,Y c ,Z c ] T is the 3D point coordinate in the camera coordinate system.

[0092] Specifically, in this embodiment, the three-dimensional point in the world coordinate system is converted into a three-dimensional coordinate in the camera coordinate system of the new viewing angle, specifically:

[0093] Get the three-dimensional point of the target area in the world coordinate system, and convert it into the three-dimensional coordinates of the new perspective in the camera coordinate system according to the camera external parameters of the new perspective; the three-dimensional coordinates of the target area in the camera coordinate system under the new perspective are:

[0094]

[0095] Among them, [X c' ,Y c' ,Z c' ] T is the 3D coordinate of the target area in the camera coordinate system under the new perspective, R B is the rotation matrix under the new perspective, t B is the displacement vector at the new viewing angle, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system.

[0096] Specifically, in this embodiment, the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area in the new viewing angle, specifically:

[0097] According to the camera intrinsic parameter matrix of the new viewing angle, the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area at the new viewing angle. The two-dimensional coordinates of the target area at the new viewing angle are:

[0098]

[0099] Among them, (u', v') is the two-dimensional coordinate of the target area at the new perspective, [X c' ,Y c' ,Z c' ] T is the 3D coordinate of the target area in the camera coordinate system under the new perspective, K B is the camera intrinsic parameter matrix under the new perspective.

[0100] Specifically, in this embodiment, the second HashNeRF model only accurately reconstructs the target area annotated by the YOLOv8 annotation model, and processes the details of the target area with higher accuracy than the first HashNeRF model. When a target area needs to be finely modeled under a new perspective, the second HashNeRF model is directly used to reconstruct the target area with high precision, combined with the occlusion and depth information provided by the depth map; the step S4 uses the second HashNeRF model to perform fine reconstruction of the target area after coordinate transformation under different perspectives, specifically: input the perspective pose under the new perspective, and the two-dimensional coordinates of the target area under the new perspective, and generate a color map and a depth map of the position coordinates under the new perspective through the second HashNeRF model.

[0101] Specifically, in this embodiment, the step S4 identifies the occluded area according to the depth maps of different viewing angles, specifically:

[0102] When switching to a different viewing angle, a global view of the target object under the viewing angle is generated through the first HashNeRF model, and the depth information under the switched viewing angle is obtained;

[0103] The three-dimensional coordinates of the target area annotated by the YOLOv8 annotation model are mapped to the switching perspective through coordinate transformation to obtain the target area under the switching perspective;

[0104] The second HashNeRF model is used to achieve fine reconstruction of the target area under the switching perspective, that is, to perform high-precision identification and restoration of the occluded area and render high-precision details of the target area.

[0105] Specifically, in this embodiment, the HashNeRF model is used to perform high-precision modeling on the target area, and refined reconstruction under different perspectives is achieved according to coordinate transformation, which is particularly suitable for the situation where the target object is partially occluded; the occluded area is identified by analyzing the depth information, and the high-precision restoration of the target area or the occluded area is completed in combination with the second HashNeRF model; the present invention cooperates with the first HashNeRF model and the second HashNeRF model, one is responsible for the reconstruction of the global scene, and the other focuses on the high-precision modeling of the target area, and the depth map generated by the first HashNeRF model and the coordinate transformation of the target area are used, and the occluded area is restored with high precision in combination with the second HashNeRF model, which significantly improves the reconstruction effect of the occluded area and overcomes the problem of insufficient reconstruction accuracy of the occluded area of ​​the traditional model; and there is no need to perform complex manual labeling or identification of the occluded area, and the occluded area is automatically detected through multi-perspective depth differences, and automatically restored in combination with the second HashNeRF model, thereby improving the automation level of occlusion processing.

[0106] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for fine-grained 3D reconstruction of target objects based on YOLOv8 annotation, characterized in that: The following steps are involved: S1. Build a YOLOv8 annotation model, and use the YOLOv8 annotation model to annotate the boundaries of the target object and identify the unobstructed area and the target area; S2. Build a first HashNeRF model, input image data from multiple perspectives, perform global 3D reconstruction of the target object through the first HashNeRF model, and output a color map and a depth map for each perspective; S3, based on the two-dimensional coordinates of the target area identified by the YOLOv8 annotation model and the depth map output by the first HashNeRF model, performing coordinate transformation of the target area; S4. Construct a second HashNeRF model, use the second HashNeRF model to perform fine reconstruction of the target area after coordinate transformation under different perspectives, and identify the occluded area according to the depth map of different perspectives, so as to complete the high-precision restoration of the target area and the three-dimensional reconstruction of the target object.

2. The method for fine-grained 3D reconstruction of target objects based on YOLOv8 annotation according to claim 1, characterized in that: The step S1 is specifically: The YOLOv8 annotation model is used to detect each frame of the input image data, generate a mask of the target area, annotate the target area in the image, and output the two-dimensional coordinates and bounding box size of the target area.

3. A method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation according to any one of claims 1 to 2, characterized in that: The step S2 is specifically: S21, inputting the three-dimensional space coordinates and viewing directions of multiple viewing angle images, and using hash coding to map the three-dimensional space coordinates to a high-dimensional feature space; S22, performing three-dimensional linear interpolation and splicing on the encoding results of different resolutions according to the characteristics of the hash table to form a feature vector containing global and local information of the scene; S23, based on the concatenated feature vector, extracting the density and color information of the scene from the high-dimensional encoding space through a multi-layer perceptron, and calculating the final color value of each ray in the scene according to the density and color information of the scene; S24. Construct a loss function, optimize the network weights of the first HashNeRF model through back propagation, realize global 3D reconstruction of the target object, and output color maps and depth maps of each perspective.

4. A method for refining 3D reconstruction of a target object based on YOLOv8 annotation according to any one of claims 1 to 2, characterized in that: The step S3 is specifically: According to the depth map of each viewing angle output by the first HashNeRF model, the depth value at the current viewing angle is obtained; Calculate the 3D points of the target area based on the 2D coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle; Convert the 3D points in the target area into 3D points in the world coordinate system according to the current camera position; The three-dimensional point in the world coordinate system is converted into the three-dimensional coordinate in the camera coordinate system of the new perspective, and the three-dimensional coordinate in the camera coordinate system is projected onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target area in the new perspective.

5. The method for fine-grained 3D reconstruction of target objects based on YOLOv8 annotation according to claim 4, characterized in that: The three-dimensional points of the target area are calculated according to the two-dimensional coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle, specifically: Get the camera intrinsic parameter matrix of the current viewing angle; the camera intrinsic parameter matrix is: Among them, K is the camera internal parameter matrix, f x and f y is the focal length of the camera, c x and c y is the coordinate of the center point of the image; According to the two-dimensional coordinates of the target area annotated by the YOLOv8 annotation model and the depth value under the corresponding viewing angle, the camera intrinsic parameter matrix is ​​used to convert the two-dimensional coordinates of the target area into three-dimensional points in the camera coordinate system, specifically: Among them, (u, v) is the two-dimensional coordinate of the target area under the current viewing angle, Z c is the depth value at the corresponding viewing angle, [X c ,Y c ,Z c ] T is the 3D point coordinate in the camera coordinate system.

6. The method for fine-grained 3D reconstruction of target objects based on YOLOv8 annotation according to claim 4, characterized in that: The three-dimensional points of the target area are converted into three-dimensional points in the world coordinate system according to the current camera position, specifically: Obtain the camera extrinsic parameters of the current viewing angle; the camera extrinsic parameters include the rotation matrix and displacement vector of the camera; According to the camera external parameters, the three-dimensional points in the target area are converted into three-dimensional points in the world coordinate system, specifically: Among them, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system, R A is the rotation matrix at the current viewing angle, t A is the displacement vector at the current viewing angle, [X c ,Y c ,Z c ] T is the 3D point coordinate in the camera coordinate system.

7. The method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation according to claim 4, characterized in that: The three-dimensional point in the world coordinate system is converted into the three-dimensional coordinate in the camera coordinate system of the new perspective, specifically: Get the three-dimensional point of the target area in the world coordinate system, and convert it into the three-dimensional coordinates of the new perspective in the camera coordinate system according to the camera external parameters of the new perspective; the three-dimensional coordinates of the target area in the camera coordinate system under the new perspective are: Among them, [X c' ,Y c' ,Z c' ] T is the 3D coordinate of the target area in the camera coordinate system under the new perspective, R B is the rotation matrix under the new perspective, t B is the displacement vector at the new viewing angle, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system.

8. The method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation according to claim 4, characterized in that: The three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area in the new viewing angle, specifically: According to the camera intrinsic parameter matrix of the new viewing angle, the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area at the new viewing angle. The two-dimensional coordinates of the target area at the new viewing angle are: Among them, (u', v') is the two-dimensional coordinate of the target area at the new perspective, [X c' ,Y c' ,Z c' ] T is the 3D coordinate of the target area in the camera coordinate system under the new perspective, K B is the camera intrinsic parameter matrix under the new perspective.

9. A method for refining 3D reconstruction of a target object based on YOLOv8 annotation according to any one of claims 1 to 2, characterized in that: The step S4 uses the second HashNeRF model to perform fine reconstruction of the target area after coordinate transformation under different viewing angles, specifically: The viewing pose under the new viewing angle and the two-dimensional coordinates of the target area under the new viewing angle are input, and the color map and depth map of the position coordinates under the new viewing angle are generated through the second HashNeRF model.

10. A method for fine-grained 3D reconstruction of a target object based on YOLOv8 annotation according to any one of claims 1-2, characterized in that: The step S4 identifies the occluded area according to the depth maps of different viewing angles, specifically: When switching to a different viewing angle, a global view of the target object under the viewing angle is generated through the first HashNeRF model, and the depth information under the switched viewing angle is obtained; The three-dimensional coordinates of the target area annotated by the YOLOv8 annotation model are mapped to the switching perspective through coordinate transformation to obtain the target area under the switching perspective; The second HashNeRF model is used to achieve fine reconstruction of the target area under the switching perspective, that is, to perform high-precision identification and restoration of the occluded area and render high-precision details of the target area.

Images