A target object fine three-dimensional reconstruction method based on YOLOv8 labeling

Abstract

The application discloses a target object fine three-dimensional reconstruction method based on YOLOv8 labeling, comprising the following steps: constructing a YOLOv8 labeling model, performing boundary labeling on a target object through the YOLOv8 labeling model, and identifying an unoccluded area and a target area; constructing a first HashNeRF model, inputting image data of multiple perspectives, performing global three-dimensional reconstruction of the target object through the first HashNeRF model, and outputting a color graph and a depth graph of each perspective; performing coordinate conversion of the target area based on the two-dimensional coordinates of the target area identified by the YOLOv8 labeling model and the depth graph output by the first HashNeRF model; constructing a second HashNeRF model, performing fine reconstruction of the target area under different perspectives through the second HashNeRF model, identifying an occluded area according to the depth graphs of different perspectives, and completing high-precision restoration of the target area and three-dimensional reconstruction of the target object. The application solves the technical problems of insufficient reconstruction accuracy of the occluded area and poor consistency of the multi-perspective target area in the existing three-dimensional scene reconstruction through the NeRF technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a target object fine three-dimensional reconstruction method based on YOLOv8 labeling. BACKGROUND

[0002] Neural Radiance Fields (NeRF) is a neural network-based three-dimensional scene reconstruction technology that can generate highly realistic three-dimensional scenes by learning the color and collective features of the scene from multiple perspective images. NeRF uses ray tracing and volume rendering techniques to integrate images from different perspectives into a unified radiance field, achieving realistic reconstruction of the scene. However, there is a key challenge in the application of NeRF, which is that when the target object is occluded at certain perspectives, the model may not be able to completely reconstruct the structure and details of that area. NeRF technology focuses on the overall reconstruction of the scene, and its model usually learns and reconstructs from a global perspective, thus ignoring the fine processing of specific target areas in the scene. Existing NeRF technology relies on multiple perspective images for three-dimensional scene reconstruction, but when certain perspectives in the input images are occluded, NeRF cannot learn complete target object information from these perspectives, resulting in incomplete reconstruction of some areas. Moreover, NeRF cannot effectively distinguish between target objects and backgrounds, leading to wasted computing resources on irrelevant background areas, affecting the modeling quality of target areas, i.e., there is background interference. In recent years, some technologies have attempted to predict the contents of occluded areas through image inference or completion techniques of neural networks, usually relying on the pixel information near the occluded area, which can repair the occluded area to some extent, but cannot utilize multi-perspective information for unified reconstruction of three-dimensional scenes. Or try to handle the occlusion problem in a single image, using semantic segmentation or deep learning to predict and repair the target area. The effectiveness of this method usually depends on the context information of a single image, making it difficult to achieve consistent completion across perspectives. It can be seen that the above technical solutions cannot fully utilize the advantages of multi-perspective images to efficiently handle the situation where the target area is occluded in a complex scene. Therefore, it is urgent to propose a target object fine three-dimensional reconstruction method based on YOLOv8 labeling to solve the technical problems of insufficient reconstruction accuracy of occluded areas and poor consistency of multi-perspective target areas in existing three-dimensional scene reconstruction using NeRF technology. SUMMARY

[0003] The main purpose of the present application is to propose a target object fine three-dimensional reconstruction method based on YOLOv8 labeling, aiming to solve the technical problems of insufficient reconstruction accuracy of occluded areas and poor consistency of multi-perspective target areas in existing three-dimensional scene reconstruction using NeRF technology.

[0004] To achieve the above object, the application provides a target object fine three-dimensional reconstruction method based on YOLOv8 labeling, wherein the target object fine three-dimensional reconstruction method based on YOLOv8 labeling comprises the following steps:

[0005] S1, a YOLOv8 labeling model is constructed, and the target object is labeled by boundary through the YOLOv8 labeling model to identify the unoccluded area and the target area;

[0006] S2, a first HashNeRF model is constructed, image data of multiple perspectives is input, global three-dimensional reconstruction of the target object is carried out through the first HashNeRF model, and color maps and depth maps of each perspective are output;

[0007] S3, based on the two-dimensional coordinates of the target area identified by the YOLOv8 labeling model and the depth maps output by the first HashNeRF model, coordinate conversion of the target area is carried out;

[0008] S4, a second HashNeRF model is constructed, fine reconstruction of the target area after coordinate conversion is carried out under different perspectives through the second HashNeRF model, and the occluded area is identified according to the depth maps of different perspectives, so that high-precision restoration of the target area and three-dimensional reconstruction of the target object are completed.

[0009] In one of the preferred schemes, the step S1 is specifically:

[0010] Each frame of image of the input image data is detected through the YOLOv8 labeling model, a mask of the target area is generated, the target area in the image is labeled, and two-dimensional coordinates and boundary box size of the target area are output.

[0011] In one of the preferred schemes, the step S2 is specifically:

[0012] S21, three-dimensional space coordinates and perspective directions of multiple perspective images are input, and the three-dimensional space coordinates are mapped to a high-dimensional feature space using hash coding;

[0013] S22, according to the features of the hash table, the coding results of different resolutions are three-dimensional linearly interpolated and spliced to form a feature vector containing global and local information of the scene;

[0014] S23, based on the spliced feature vector, the density and color information of the scene are extracted from the high-dimensional coding space through a multilayer perceptron, and the final color value of each ray in the scene is calculated according to the density and color information of the scene;

[0015] S24, a loss function is constructed, the network weight of the first HashNeRF model is optimized through back propagation, global three-dimensional reconstruction of the target object is realized, and color maps and depth maps of each perspective are output.

[0016] In one preferred embodiment, the step S3 specifically comprises:

[0017] According to the depth map of each view output by the first HashNeRF model, the depth value under the current view is obtained.

[0018] According to the two-dimensional coordinates of the target region labeled by the YOLOv8 labeling model and the depth value under the corresponding view, the three-dimensional point of the target region is calculated.

[0019] According to the camera pose of the current view, the three-dimensional point of the target region is converted into a three-dimensional point in the world coordinate system.

[0020] The three-dimensional point in the world coordinate system is converted into a three-dimensional coordinate in the camera coordinate system of the new view, and the three-dimensional coordinate in the camera coordinate system is projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target region under the new view.

[0021] In one preferred embodiment, the three-dimensional point of the target region is calculated according to the two-dimensional coordinates of the target region labeled by the YOLOv8 labeling model and the depth value under the corresponding view, specifically:

[0022] The camera intrinsic matrix of the current view is obtained; the camera intrinsic matrix is:

[0023]

[0024] where K is the camera intrinsic matrix, f x and f y are the focal lengths, c x and c y are the image center point coordinates.

[0025] According to the two-dimensional coordinates of the target region labeled by the YOLOv8 labeling model and the depth value under the corresponding view, the two-dimensional coordinates of the target region are converted into a three-dimensional point in the camera coordinate system using the camera intrinsic matrix, specifically:

[0026]

[0027] where (u, v) are the two-dimensional coordinates of the target region under the current view, Z c is the depth value under the corresponding view, [X c , Y c , Z c ] T is the three-dimensional point coordinate in the camera coordinate system.

[0028] In one preferred embodiment, the three-dimensional point of the target region is converted into a three-dimensional point in the world coordinate system according to the camera pose of the current view, specifically:

[0029] obtaining camera extrinsic parameters of the current view angle; the camera extrinsic parameters include a rotation matrix and a displacement vector of the camera;

[0030] converting the three-dimensional point of the target region into a three-dimensional point in the world coordinate system according to the camera extrinsic parameters, specifically:

[0031]

[0032] wherein, [X w ,Y w ,Z w ] T is the three-dimensional point in the world coordinate system, R A is the rotation matrix of the current view angle, t A is the displacement vector of the current view angle, [X c ,Y c ,Z c ] T is the three-dimensional point coordinate in the camera coordinate system.

[0033] In one of the preferred solutions, the conversion of the three-dimensional point in the world coordinate system into a three-dimensional coordinate in the camera coordinate system of the new view angle is specifically:

[0034] obtaining the three-dimensional point of the target region in the world coordinate system, and converting it into a three-dimensional coordinate in the camera coordinate system of the new view angle according to the camera extrinsic parameters of the new view angle; the three-dimensional coordinate of the target region in the camera coordinate system of the new view angle is:

[0035]

[0036] wherein, [X c' ,Y c' ,Z c' ] T is the three-dimensional coordinate of the target region in the camera coordinate system of the new view angle, R B is the rotation matrix of the new view angle, t B is the displacement vector of the new view angle, [X w ,Y w ,Z w ] T is the three-dimensional point in the world coordinate system.

[0037] In one of the preferred solutions, the projection of the three-dimensional coordinate in the camera coordinate system onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target region in the new view angle is specifically:

[0038] projecting the three-dimensional coordinate in the camera coordinate system onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target region in the new view angle according to the camera intrinsic parameter matrix of the new view angle, the two-dimensional coordinate of the target region in the new view angle is:

[0039]

[0040] wherein (u', v') is the two-dimensional coordinates of the target region under the new view angle, [X c' ,Y c' ,Z c' ] T are the three-dimensional coordinates of the target region under the new view angle in the camera coordinate system, K B is the camera intrinsic matrix under the new view angle.

[0041] In one preferred embodiment, the step S4 performs fine reconstruction of the target region under different view angles through the second HashNeRF model, specifically:

[0042] The view pose under the new view angle and the two-dimensional coordinates of the target region under the new view angle are input, and the color map and the depth map of the position coordinates under the new view angle are generated through the second HashNeRF model.

[0043] In one preferred embodiment, the step S4 identifies the occluded region according to the depth map under different view angles, specifically:

[0044] When switching to a different view angle, the global view of the target object under the view angle is generated through the first HashNeRF model, and the depth information under the switched view angle is obtained;

[0045] The three-dimensional coordinates of the target region labeled by the YOLOv8 labeling model are mapped to the switched view angle through coordinate conversion, and the target region under the switched view angle is obtained;

[0046] The fine reconstruction of the target region under the switched view angle is realized through the second HashNeRF model, that is, the high-precision identification and restoration of the occluded region are realized, and the high-precision details of the target region are rendered.

[0047] In the technical scheme of the present application, the target object fine three-dimensional reconstruction method based on YOLOv8 labeling includes the following steps: constructing a YOLOv8 labeling model and performing boundary labeling on the target object through the YOLOv8 labeling model to identify unoccluded areas and target areas; constructing a first HashNeRF model, inputting image data of multiple perspectives, performing global three-dimensional reconstruction of the target object through the first HashNeRF model, and outputting color maps and depth maps of each perspective; based on the two-dimensional coordinates of the target area identified by the YOLOv8 labeling model and the depth map output by the first HashNeRF model, the coordinates of the target area are converted; a second HashNeRF model is constructed, the target area after coordinate conversion is reconstructed in different perspectives through the second HashNeRF model, and the occluded area is identified according to the depth map of different perspectives, so as to complete high-precision restoration of the target area and three-dimensional reconstruction of the target object. The present application solves the technical problems of insufficient reconstruction accuracy of the occluded area and poor consistency of the multi-perspective target area in the existing three-dimensional scene reconstruction by NeRF technology.

[0048] In the present application, by setting the first HashNeRF model and the second HashNeRF model, the global scene is reconstructed by the first HashNeRF model, and when rendering the scene, the second HashNeRF model uses the depth information provided by the first HashNeRF model to finely reconstruct the target area in a specific perspective, which ensures the integrity of the global scene and realizes high-precision detail presentation in the local area.

[0049] In the present application, the depth map generated by the first HashNeRF model is combined with the target area labeled by the YOLOv8 labeling model, and the coordinates of the target area are directly mapped to different perspectives without relying on the global model to speculate and infer the occluded area, which improves the reconstruction accuracy of the occluded area and overcomes the shortcomings of the prior art that relies on the global model to infer the occluded area, resulting in inaccurate reconstruction. Combined with the depth map and camera pose information, accurate coordinate conversion of the target area can be realized, and the reconstruction results under different perspectives are more coordinated and consistent.

[0050] In the present application, the second HashNeRF model focuses on learning and reconstructing the target area, and provides stronger detail recovery capability when the occluded area is re-observed in combination with the target area labeled by the YOLOv8 labeling model, thereby avoiding the defect that the occluded area is not recovered finely enough, improving the detail recovery capability of the occluded area, and at the same time, dividing the reconstruction task into global reconstruction and target area fine reconstruction, reducing unnecessary global detail processing burden, thereby improving the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some of the embodiments of the present application, and other drawings can also be obtained according to the structures shown in the drawings without creative labor for those skilled in the art.

[0052] Figure 1 A schematic diagram of a target object fine three-dimensional reconstruction method based on YOLOv8 labeling according to an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a YOLOv8 labeling model according to an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a first HashNeRF model according to an embodiment of the present application.

[0055] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0057] In addition, the description such as "first", "second" and the like in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features.

[0058] In addition, the technical solutions of the various embodiments of the present application can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist, and is not within the scope of protection required by the present application.

[0059] Referring to Figure 1 According to an aspect of the present application, a target object fine three-dimensional reconstruction method based on YOLOv8 labeling is provided, wherein the target object fine three-dimensional reconstruction method based on YOLOv8 labeling comprises the following steps:

[0060] S1, a YOLOv8 labeling model is constructed, and a target object is labeled with a boundary through the YOLOv8 labeling model to identify an unoccluded area and a target area;

[0061] S2, a first HashNeRF model is constructed, image data of multiple perspectives is input, global three-dimensional reconstruction of a target object is performed through the first HashNeRF model, and a color map and a depth map of each perspective are output;

[0062] S3, based on the two-dimensional coordinates of the target area identified by the YOLOv8 labeling model and the depth map output by the first HashNeRF model, coordinate conversion of the target area is performed;

[0063] S4, a second HashNeRF model is constructed, the target area after coordinate conversion is reconstructed in different perspectives through the second HashNeRF model, and an occluded area is identified according to the depth map of different perspectives, so as to complete high-precision restoration of the target area and three-dimensional reconstruction of the target object.

[0064] Specifically, in the embodiment, referring to Figure 2 is a schematic diagram of the YOLOv8 labeling model, each frame of image of the input image data is detected through the YOLOv8 labeling model, and a mask of the target area is generated, the target area in the image is labeled, and two-dimensional coordinates and a bounding box size of the target area are output; the target area of the input multiple frames of image is identified through the YOLOv8 labeling model, two-dimensional coordinates, a bounding box or a mask of the target object are output, and labeling and tracking are performed in multiple perspectives, which provides basic data for subsequent fine reconstruction.

[0065] Specifically, in the embodiment, referring to Figure 3 is a network structure of the first HashNeRF model, the target object is reconstructed in a global three-dimensional scene through the first HashNeRF model; the first HashNeRF model is an improved network based on neural radiation field, which is used for efficient three-dimensional scene reconstruction and rendering, and through the combination of hash coding volume rendering, the training and calculation speed is greatly improved while maintaining high rendering quality.

[0066] Specifically, in the embodiment, the step S2 is specifically:

[0067] S21, input the three-dimensional space coordinates and the perspective direction of the multiple perspective images, and map the three-dimensional space coordinates to a high-dimensional feature space using hash coding; for the input three-dimensional space coordinates, encode through hash tables with different resolutions to form multi-scale space features; these encoders are similar to the position information encoding in traditional neural radiance fields, but through the sparse representation of hash tables, the memory occupation and calculation overhead are greatly reduced; each layer in the hash table corresponds to a different resolution, providing multi-level detail capture capability from low resolution to high resolution, which helps to model local details in complex scenes;

[0068] S22, use the features of the multiple hash tables to perform three-dimensional linear interpolation and splicing on the encoding results with different resolutions to form a feature vector containing global and local information of the scene; the feature vector retains the global and local information of the scene and can capture complex geometric structures and texture changes;

[0069] S23, based on the spliced feature vector, extract the density and color information of the scene from the high-dimensional encoding space through a multi-layer perception, and calculate the final color value of each ray in the scene according to the density and color information of the scene; wherein, the density prediction, a part of the multi-layer perception outputs the volume density, which is used to determine the position of the obstacle encountered by the ray in the scene; the color prediction, another part of the multi-layer perception combines the perspective direction to calculate the color value of the voxel, so that the scene at different perspectives is more realistic; wherein, the final color value of each ray in the scene is calculated using a volume rendering formula according to the density and color information of the scene, specifically: the ray passes through the three-dimensional scene, and the density value is integrated along the ray direction to accumulate the contribution of each voxel; the color value is obtained by weighted sum of the color value of each voxel, and the weight depends on the density of the voxel and the transmittance of the ray at the voxel;

[0070] S24, construct a loss function, optimize the network weights of the first HashNeRF model through back propagation, realize global three-dimensional reconstruction of the target object, and output color maps and depth maps of each perspective; specifically, during the training process of the first HashNeRF model, the difference between the synthesized 2D image and the actual 2D observation image is used as a loss function, the network weights of the first HashNeRF model are optimized through back propagation, and through the combination of hash coding and neural radiance field, the model can effectively learn the three-dimensional structure of a complex scene with less computing resources and time.

[0071] Specifically, in the present embodiment, the first HashNeRF model can perform query and difference operations. Compared with traditional models, the first HashNeRF model avoids unnecessary calculations through sparse representation and multi-resolution features, thereby ensuring a significant acceleration of rendering speed while maintaining accuracy. Meanwhile, the first HashNeRF model efficiently represents space through multi-resolution hash coding, and performs density and color prediction of the scene through a lightweight multi-layer perceptron. Finally, high-quality image generation is achieved through rendering, i.e., global three-dimensional reconstruction of the target object, ensuring that the first HashNeRF model can achieve efficient rendering and high-fidelity scene reconstruction in complex scenes, thereby reducing the complexity of calculations.

[0072] Specifically, in the present embodiment, the first HashNeRF model and the second HashNeRF model have the same network structure. After training, the first HashNeRF model and the second HashNeRF model obtain the color map and the depth map of the position target under the perspective by inputting the perspective information and the position coordinates under the perspective. The network structure of the first HashNeRF model and the second HashNeRF model is described with reference to Figure 3 .

[0073] Specifically, in the present embodiment, the first HashNeRF model is used to perform global reconstruction on multi-perspective images to generate a color map and a depth map of the scene. The first HashNeRF model provides global three-dimensional structure and depth information under each perspective as the geometric basis for fine reconstruction. The two-dimensional coordinates labeled by the YOLOv8 labeling model, the depth map generated by global three-dimensional reconstruction, and the camera pose under the corresponding perspective are used to convert the two-dimensional coordinates of the target region into three-dimensional coordinates under different perspectives to accurately locate the position of the target region in multiple perspectives. The two-dimensional coordinates labeled by the YOLOv8 labeling model are converted into three-dimensional space coordinates through the depth map and camera pose information of multiple perspectives, thereby realizing reconstruction under different perspectives and enhancing the compatibility and coordination between perspectives.

[0074] Specifically, in the present embodiment, the step S3 specifically includes:

[0075] According to the depth value under the current perspective obtained from the depth map of each perspective output by the first HashNeRF model;

[0076] According to the two-dimensional coordinates of the target region labeled by the YOLOv8 labeling model and the depth value under the corresponding perspective, the three-dimensional points of the target region are calculated;

[0077] According to the camera pose of the current perspective, the three-dimensional points of the target region are converted into three-dimensional points in the world coordinate system;

[0078] The three-dimensional point in the world coordinate system is converted into a three-dimensional coordinate in the camera coordinate system of the new view angle, and the three-dimensional coordinate in the camera coordinate system is projected onto a two-dimensional image plane to obtain a two-dimensional coordinate of the target region under the new view angle.

[0079] Specifically, in the embodiment, the two-dimensional coordinates of the target region and the depth values under the corresponding view angle labeled according to the YOLOv8 labeling model are used to calculate the three-dimensional point of the target region, specifically:

[0080] The camera intrinsic matrix of the current view angle is obtained; the camera intrinsic matrix is:

[0081]

[0082] wherein K is the camera intrinsic matrix, f x and f y are the focal lengths of the camera, c x and c y are the image center points.

[0083] The two-dimensional coordinates of the target region are converted into three-dimensional points in the camera coordinate system by using the camera intrinsic matrix according to the two-dimensional coordinates of the target region and the depth values under the corresponding view angle labeled by the YOLOv8 labeling model, specifically:

[0084]

[0085] wherein (u, v) is the two-dimensional coordinate of the target region under the current view angle, Z c is the depth value under the corresponding view angle, [X c , Y c , Z c ] T is the three-dimensional point coordinate in the camera coordinate system.

[0086] Specifically, in the embodiment, the three-dimensional point of the target region is converted into a three-dimensional point in the world coordinate system according to the camera pose of the current view angle, specifically:

[0087] The camera extrinsic of the current view angle is obtained; the camera extrinsic includes the rotation matrix R and the displacement vector t of the camera; the camera extrinsic is the description of the camera in the world coordinate system, that is, the rotation matrix and the unique vector of the camera, which is used to convert the point in the world coordinate system to the camera coordinate system; specifically:

[0088] [X c , Y c , Z c ] T = R[X w , Y w , Z w ] T + t

[0089] According to the camera extrinsic parameter, the three-dimensional point of the target region is converted into a three-dimensional point in the world coordinate system, specifically:

[0090]

[0091] wherein, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system, R A is a rotation matrix in the current view angle, t A is a displacement vector in the current view angle, [X c ,Y c ,Z c ] T is a three-dimensional point coordinate in the camera coordinate system.

[0092] Specifically, in the embodiment, the three-dimensional point in the world coordinate system is converted into a three-dimensional coordinate in the camera coordinate system of the new view angle, specifically:

[0093] The three-dimensional point of the target region in the world coordinate system is obtained, and according to the camera extrinsic parameter of the new view angle, it is converted into a three-dimensional coordinate in the camera coordinate system of the new view angle; the three-dimensional coordinate of the target region in the camera coordinate system of the new view angle is:

[0094]

[0095] wherein, [X c' ,Y c' ,Z c' ] T is a three-dimensional coordinate of the target region in the camera coordinate system of the new view angle, R B is a rotation matrix in the new view angle, t B is a displacement vector in the new view angle, [X w ,Y w ,Z w ] T is a three-dimensional point in the world coordinate system.

[0096] Specifically, in the embodiment, the three-dimensional coordinate in the camera coordinate system is projected onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target region in the new view angle, specifically:

[0097] According to the camera intrinsic parameter matrix of the new view angle, the three-dimensional coordinate in the camera coordinate system is projected onto the two-dimensional image plane to obtain the two-dimensional coordinate of the target region in the new view angle, and the two-dimensional coordinate of the target region in the new view angle is:

[0098]

[0099] wherein (u', v') is the two-dimensional coordinates of the target region under the new view angle, [X c' ,Y c' ,Z c' ] T is the three-dimensional coordinates of the target region under the new view angle in the camera coordinate system, K B is the camera intrinsic matrix under the new view angle.

[0100] Specifically, in the present embodiment, the second HashNeRF model is used to precisely reconstruct the target region labeled by the YOLOv8 labeling model, and the details of the target region are processed by the second HashNeRF model with higher precision than the first HashNeRF model. When fine modeling of a target region under a new view angle is required, the second HashNeRF model is directly used to reconstruct the target region with high precision, and the occlusion and depth information provided by the depth map are combined. The step S4 precisely reconstructs the target region under different view angles through the second HashNeRF model after coordinate conversion, specifically: the view pose under the new view angle and the two-dimensional coordinates of the target region under the new view angle are input, and the color map and the depth map of the position coordinates under the new view angle are generated through the second HashNeRF model.

[0101] Specifically, in the present embodiment, the step S4 identifies the occlusion region according to the depth map under different view angles, specifically:

[0102] When switching to a different view angle, the first HashNeRF model is used to generate a global view of the target object under the view angle, and the depth information under the switched view angle is obtained.

[0103] The three-dimensional coordinates of the target region labeled by the YOLOv8 labeling model are mapped to the switched view angle through coordinate conversion, and the target region under the switched view angle is obtained.

[0104] The second HashNeRF model is used to precisely reconstruct the target region under the switched view angle, that is, to precisely identify and restore the occlusion region, and to render the high-precision details of the target region.

[0105] Specifically, in the embodiment, the HashNeRF model is used to model the target area with high precision, and the fine reconstruction under different perspectives is realized through coordinate transformation, which is especially suitable for the case that the target object is partially occluded; through the analysis of depth information, the occluded area is identified, and the high-precision restoration of the target area or the occluded area is completed in combination with the second HashNeRF model; the first HashNeRF model and the second HashNeRF model work together in the present application, one is responsible for the reconstruction of the global scene, and the other focuses on the high-precision modeling of the target area; the depth map generated by the first HashNeRF model and the coordinate transformation of the target area are combined to restore the occluded area with high precision by the second HashNeRF model, which significantly improves the reconstruction effect of the occluded area and overcomes the problem of insufficient reconstruction accuracy of the traditional model in the occluded area; and without complex manual labeling or identification of the occluded area, the occluded area is automatically detected through the multi-perspective depth difference, and the second HashNeRF model is combined for automatic restoration, which improves the automation level of the occlusion processing.

[0106] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or directly / indirectly applied in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for refined 3D reconstruction of target objects based on YOLOv8 annotation, characterized in that, Includes the following steps: S1. Construct a YOLOv8 annotation model and use the YOLOv8 annotation model to annotate the boundaries of the target object, and identify the unobstructed area and the target area. S2. Construct the first HashNeRF model. Input image data from multiple perspectives, perform global 3D reconstruction of the target object using the first HashNeRF model, and output color maps and depth maps for each perspective; specifically: S21. Input the three-dimensional spatial coordinates and viewpoint direction of multiple viewpoint images, and use hash encoding to map the three-dimensional spatial coordinates to a high-dimensional feature space; S22. Based on the characteristics of the hash table, perform three-dimensional linear interpolation and splicing on the encoding results of different resolutions to form a feature vector containing global and local information of the scene. S23. Based on the concatenated feature vectors, the density and color information of the scene are extracted from the high-dimensional coding space through a multilayer perceptron, and the final color value of each ray in the scene is calculated according to the density and color information of the scene. S24. Construct a loss function, optimize the network weights of the first HashNeRF model through backpropagation, realize the global 3D reconstruction of the target object, and output the color map and depth map of each view. S3. Based on the two-dimensional coordinates of the target region identified by the YOLOv8 annotation model and the depth map output by the first HashNeRF model, perform coordinate transformation of the target region; specifically: Based on the depth maps of each viewpoint output by the first HashNeRF model, obtain the depth value at the current viewpoint; Calculate the three-dimensional points of the target region based on the two-dimensional coordinates of the target region annotated by the YOLOv8 annotation model and the depth value at the corresponding viewpoint. Based on the current camera pose and viewpoint, convert the 3D points of the target area into 3D points in the world coordinate system. The three-dimensional points in the world coordinate system are converted into three-dimensional coordinates in the camera coordinate system of the new perspective, and the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area in the new perspective. S4. Construct a second HashNeRF model, and use the second HashNeRF model to perform refined reconstruction of the target area after coordinate transformation from different perspectives. Identify occluded areas based on the depth maps from different perspectives, thereby completing the high-precision restoration of the target area and the three-dimensional reconstruction of the target object. The detailed reconstruction of the target area after coordinate transformation using the second HashNeRF model under different perspectives is specifically as follows: input the viewpoint pose under the new perspective and the two-dimensional coordinates of the target area under the new perspective, and generate a color map and depth map of the position coordinates under the new perspective using the second HashNeRF model. The process of identifying occluded regions based on depth maps from different viewpoints specifically involves: When switching to different viewpoints, a global view of the target object in that viewpoint is generated using the first HashNeRF model, and depth information in the switched viewpoint is obtained; The 3D coordinates of the target area annotated by the YOLOv8 annotation model are mapped to the changed viewpoint through coordinate transformation to obtain the target area under the changed viewpoint; The second HashNeRF model is used to achieve fine reconstruction of the target area under the switching perspective, that is, to perform high-precision identification and restoration of occluded areas and render high-precision target area details.

2. The method for refined 3D reconstruction of target objects based on YOLOv8 annotation according to claim 1, characterized in that, Step S1 specifically includes: The YOLOv8 annotation model is used to detect each frame of the input image data, generate a mask of the target region, annotate the target region in the image, and output the two-dimensional coordinates and bounding box size of the target region.

3. A method for refined 3D reconstruction of target objects based on YOLOv8 annotation according to any one of claims 1-2, characterized in that, The calculation of the three-dimensional points of the target region based on the two-dimensional coordinates of the target region annotated by the YOLOv8 annotation model and the depth value at the corresponding viewpoint is specifically as follows: Obtain the camera intrinsic parameter matrix for the current viewpoint; the camera intrinsic parameter matrix is: Where is the camera intrinsic parameter matrix, is the camera focal length, and is the coordinates of the image center point; Based on the 2D coordinates of the target region annotated by the YOLOv8 annotation model and the corresponding depth values ​​from the viewpoint, the 2D coordinates of the target region are converted into 3D points in the camera coordinate system using the camera intrinsic parameter matrix. Specifically: Where is the two-dimensional coordinate of the target area from the current viewpoint, is the depth value from the corresponding viewpoint, and is the three-dimensional point coordinate in the camera coordinate system.

4. A method for refined 3D reconstruction of target objects based on YOLOv8 annotation according to any one of claims 1-2, characterized in that, The process of converting the 3D points of the target area into 3D points in the world coordinate system based on the current camera pose is as follows: Obtain the camera extrinsic parameters of the current viewpoint; the camera extrinsic parameters include the camera's rotation matrix and translation vector; Based on the camera extrinsic parameters, the 3D points of the target area are converted into 3D points in the world coordinate system, specifically as follows: Where is a 3D point in the world coordinate system, is the rotation matrix in the current viewpoint, is the displacement vector in the current viewpoint, and is the coordinates of the 3D point in the camera coordinate system.

5. A method for refined 3D reconstruction of target objects based on YOLOv8 annotation according to any one of claims 1-2, characterized in that, The process of converting 3D points in the world coordinate system to 3D coordinates in the camera coordinate system of the new perspective specifically involves: Obtain the 3D points of the target area in the world coordinate system, and convert them into 3D coordinates in the camera coordinate system based on the camera extrinsic parameters of the new viewpoint; the 3D coordinates of the target area in the camera coordinate system under the new viewpoint are: Where is the three-dimensional coordinate of the target area in the camera coordinate system under the new perspective, is the rotation matrix under the new perspective, is the displacement vector under the new perspective, and is the three-dimensional point in the world coordinate system.

6. A method for refined 3D reconstruction of target objects based on YOLOv8 annotation according to any one of claims 1-2, characterized in that, The process of projecting the three-dimensional coordinates in the camera coordinate system onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target area from the new perspective is specifically as follows: Based on the camera intrinsic parameter matrix of the new perspective, the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional image plane to obtain the two-dimensional coordinates of the target region under the new perspective. The two-dimensional coordinates of the target region under the new perspective are: Where is the two-dimensional coordinate of the target region under the new perspective, is the three-dimensional coordinate of the target region in the camera coordinate system under the new perspective, and is the camera intrinsic parameter matrix under the new perspective.

Images