A 3D model reconstruction method and system for local dark and weak texture scenes

Abstract

The present application provides a method and system for reconstructing a three-dimensional model for a scene with local dim textures, the method comprising: inputting a two-dimensional color image set into a pre-trained neural network structure model to obtain the corresponding position and type of each target object in each frame image; performing feature point detection on the two-dimensional color image set, determining the rotation matrix and translation vector corresponding to each frame image, and constructing an initial three-dimensional point cloud model corresponding to the two-dimensional color image set; determining the corresponding mapping range of each target object in the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image; adjusting the coordinates of each point in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model; and performing grid reconstruction on the three-dimensional point cloud model to obtain a corresponding three-dimensional grid model, thereby improving the success rate and accuracy of three-dimensional model reconstruction for scenes with local dim textures.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional model reconstruction and machine learning technology, and in particular to a three-dimensional model reconstruction method and system for a scene with local dark and weak textures. Background Art

[0002] The current 3D reconstruction technology is mainly divided into two categories: reconstruction based on traditional visual geometry and reconstruction based on machine learning. Among them, the reconstruction based on traditional visual geometry detects and matches the feature points of images at different angles of the same scene to restore the corresponding 3D point cloud of the scene and then generate a surface mesh to obtain a reconstructed 3D mesh model. This type of technology can perform non-real-time model reconstruction by combining the acquired 2D image with the laser point cloud and IMU data. The reconstruction based on machine learning is trained with a large number of data sets to extract the implicit features of the image or laser point cloud and the correspondence between these features and the 3D model, generate network weights, and then use the network weights to calculate the reconstructed model for the acquired 2D image, laser point cloud and other data. This type of technology does not need to explicitly express and calculate the specific mathematical relationship between the input data and the reconstruction model. It only needs to design a good network model, input a large amount of training data into the model to train the network weights, and then obtain the reconstructed 3D mesh model.

[0003] The reconstruction technology based on traditional visual geometry is highly dependent on the texture features of the scene and is greatly affected by image noise. When the image is too dark or the scene has weak texture, feature points are easily lost or mismatched, resulting in distortion of the reconstruction model or even reconstruction failure. The point cloud generated by the laser point cloud under strong reflective materials is far away from the actual position, causing the reconstruction model to deform, and the point cloud density determines the degree of detail of the reconstruction. The reconstruction of complex scenes requires a lot of storage space. The reconstruction based on machine learning requires a large number of scenes with different lighting, texture and other characteristics as training data, and the network weights obtained by training have no corresponding theoretical formula reasoning support. They are easily affected by factors such as the number, distribution, and type of training data, resulting in a certain deviation between the obtained network weights and the required weights, and poor generalization. Summary of the invention

[0004] In response to the above technical problems, the present application provides a three-dimensional model reconstruction method and system for local dark and weak texture scenes, which combines traditional visual geometry and neural networks to reconstruct the three-dimensional mesh model of local dark and weak texture scenes, thereby improving the success rate and accuracy of three-dimensional model reconstruction of local dark and weak texture scenes.

[0005] In a first aspect, the present application provides a 3D model reconstruction method for a local dark and weak texture scene, comprising:

[0006] Acquire an original two-dimensional color image set and perform brightness and chromaticity normalization adjustment on the original two-dimensional color image set to obtain a corresponding two-dimensional color image set;

[0007] Inputting the two-dimensional color image set into a pre-trained neural network structure model, so that the neural network structure model can identify a number of target objects in the two-dimensional color image set, and obtain the corresponding position and type of each of the target objects in each frame image;

[0008] Performing feature point detection on the two-dimensional color image set, determining the rotation matrix and translation vector corresponding to each frame image, and constructing an initial three-dimensional point cloud model corresponding to the two-dimensional color image set;

[0009] According to the rotation matrix and translation vector corresponding to each frame image, the corresponding position of each target object in each frame image is mapped to the initial three-dimensional point cloud model, thereby determining the corresponding mapping range of each target object in the initial three-dimensional point cloud model;

[0010] According to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model, the coordinates of each point in the initial three-dimensional point cloud model are adjusted to obtain a three-dimensional point cloud model;

[0011] The three-dimensional point cloud model is meshed to obtain a corresponding three-dimensional mesh model.

[0012] The embodiment of the present application provides a three-dimensional model reconstruction method for a local dark and weak texture scene. First, the brightness difference of images taken at adjacent angles of the same scene is eliminated by adjusting the brightness and chromaticity normalization of the original two-dimensional color image set, which can not only improve the detection amount of feature points and reduce the failure rate of reconstruction, but also, different from the traditional texture fusion technology, avoids the phenomenon of scene breakage caused by the inconsistent brightness caused by the use of adjacent images by adjacent grids after texture mapping of the reconstructed grid model, and solves the problem of unnatural texture transition. Further, the two-dimensional color image set is detected by the neural network structure model to identify several target objects in the image, which provides data support for the subsequent three-dimensional grid reconstruction. The target detection by the neural network structure model also solves the problem of partial missing or even reconstruction failure of the reconstruction model caused by unclear images, gray, weak features, etc. to a certain extent. Then, according to each rotation matrix and translation vector, the corresponding mapping range of each target object in the initial three-dimensional point cloud model is determined, and then the point coordinates within the mapping range are adjusted according to the type of the target object, the point cloud position is adjusted, and the problem of laser point offset caused by reflective materials is solved, and the success rate and accuracy of reconstruction are further increased. Finally, the 3D point cloud model is meshed and reconstructed to obtain a corresponding 3D mesh model, realizing a 3D reconstruction method combining traditional visual geometry and neural network technology, thereby improving the success rate and accuracy of 3D model reconstruction for local dark and weak texture scenes.

[0013] In a possible implementation, the neural network structure model is constructed based on the neural network structure model of Yolov8n and is obtained through training with a number of training data, including:

[0014] Determining a plurality of target types to be identified based on the original two-dimensional color image set;

[0015] Acquiring a plurality of two-dimensional color image data containing the target type;

[0016] Performing brightness and chromaticity normalization adjustment on each of the two-dimensional color image data to obtain a corresponding training data set;

[0017] Construct an initial neural network structure model based on Yolov8n;

[0018] The initial neural network structure model is trained to detect network weight coefficients using the training data set, and each weight coefficient in the initial neural network structure model is determined to obtain the neural network structure model.

[0019] The embodiment of the present application provides a method for training a neural network structure model. First, according to the content of an original two-dimensional color image set, several target types to be identified are determined, such as cars, bicycles, electric vehicles, flat walls, windows, doors, stairs and other target types. Then, according to the determined target types, several two-dimensional color image data are collected and brightness and chromaticity normalization adjustments are performed as corresponding training data sets. Specifically, the brightness and chromaticity normalization adjustments are performed on the two-dimensional color image data, so that the initial neural network structure model adapts to the preprocessed image data in advance during the training process, thereby improving the accuracy of the model in recognizing the content of the two-dimensional color image set.

[0020] Furthermore, the performing of feature point detection on the two-dimensional color image set, determining the rotation matrix and translation vector corresponding to each frame image, and constructing an initial three-dimensional point cloud model corresponding to the two-dimensional color image set includes:

[0021] Obtaining parameters of a camera used to capture the two-dimensional color image set;

[0022] Performing scale-invariant feature point detection on the two-dimensional color image set frame by frame to obtain a feature point set corresponding to each frame image;

[0023] Perform feature matching on each feature point of each frame image with each feature point of the next frame image, and calculate the rotation matrix and translation vector corresponding to each frame image according to the feature matching results;

[0024] According to the camera parameters and the feature point set corresponding to each frame image, the rotation matrix and the translation vector, respectively calculate and obtain the three-dimensional coordinate point set corresponding to the feature point set of each frame image;

[0025] The initial three-dimensional point cloud model is constructed based on each of the three-dimensional coordinate point sets.

[0026] The embodiment of the present application provides a method for detecting feature points of a two-dimensional color image set, and performs scale-invariant feature point detection on the two-dimensional color image set frame by frame. Considering that the two-dimensional color image set is mainly composed of continuous video frames, and there is a correlation between the previous and next frames, the embodiment of the present application adopts a solution of comparing and matching feature points of the previous and next frames, and performs feature matching on the feature points of the detected adjacent frame images in pairs, thereby obtaining the rotation matrix and translation vector corresponding to each frame image, as the basis for mapping from two dimensions to three dimensions. Then, the three-dimensional coordinate point set of each frame image is calculated according to the camera parameters and the feature point set, rotation matrix and translation vector corresponding to each frame image, and then the initial three-dimensional point cloud model is constructed according to each three-dimensional coordinate point set, and the initial three-dimensional point cloud model contains all the feature points in the two-dimensional color image set after being mapped to the three-dimensional space, providing a data basis for subsequent three-dimensional reconstruction.

[0027] Furthermore, mapping the corresponding position of each target object in each frame image to the initial three-dimensional point cloud model according to the rotation matrix and the translation vector corresponding to each frame image, and then determining the corresponding mapping range of each target object in the initial three-dimensional point cloud model, includes:

[0028] According to the rotation matrix and translation vector corresponding to each frame image, the three-dimensional coordinates of the corresponding position of each target object in each frame image after mapping are calculated, and then the initial three-dimensional mapping point set corresponding to each target object is constructed;

[0029] According to a preset screening rule, coordinate screening is performed on the initial three-dimensional mapping point set corresponding to each of the target objects to obtain the three-dimensional mapping point set corresponding to each of the target objects;

[0030] According to each of the three-dimensional mapping point sets, a mapping range corresponding to each of the target objects in the initial three-dimensional point cloud model is determined.

[0031] In the embodiment of the present application, the three-dimensional coordinates of the corresponding positions of each target object in each frame image after mapping are calculated by rotation matrix and translation vector, that is, the position range of each target object in the two-dimensional image is mapped to the three-dimensional space in a point-to-point form, and the initial three-dimensional mapping point set corresponding to each target object is constructed. Similarly, considering that the two-dimensional color image set is mainly continuous video frames, the same object will appear repeatedly in the images of different frames, so multiple position ranges will be repeatedly mapped to the three-dimensional space. Therefore, it is necessary to screen each initial three-dimensional mapping point set in the future to determine the three-dimensional mapping point set after screening for each target object. Finally, the corresponding mapping range of each target object in the initial three-dimensional point cloud model can be determined according to each three-dimensional mapping point set, and the points within a certain mapping range in the initial three-dimensional point cloud model are the points formed after the corresponding target object is mapped from the two-dimensional space to the three-dimensional space.

[0032] In a possible implementation, adjusting the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model includes:

[0033] According to the type of each target object, extracting corresponding three-dimensional mesh object models from a plurality of preset three-dimensional mesh object models;

[0034] For any first target object among the target objects, adjusting the coordinates of each point in the corresponding first mapping range in the initial three-dimensional point cloud model according to each vertex position and triangle face position in the first three-dimensional mesh object model corresponding to the first target object, so as to make the coordinates of each point in the first mapping range more closely fit the first three-dimensional mesh object model;

[0035] After the coordinate adjustment is completed, the three-dimensional point cloud model is obtained.

[0036] In the embodiment of the present application, the coordinates of each point in each mapping range in the initial three-dimensional point cloud model are adjusted according to the type of target object corresponding to each mapping range. First, it is necessary to determine each three-dimensional mesh object model of the same type as each target object for reference. Then, according to each three-dimensional mesh object model, the coordinates of each point in each mapping range are adjusted respectively, so that the coordinates of each point are more closely matched to the corresponding three-dimensional mesh object model, which solves the problem of laser point offset caused by reflective materials and further increases the success rate and accuracy of three-dimensional reconstruction.

[0037] Furthermore, the meshing of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model includes:

[0038] Triangulating and segmenting the three-dimensional point cloud model, and converting the three-dimensional point cloud model into a set consisting of a plurality of vertices, tetrahedrons and triangular faces;

[0039] Calculate the weight of each vertex according to the preset formula;

[0040] According to each of the weights and the orientation of each of the triangular faces, the set is segmented using maximum flow minimum cut to generate the three-dimensional mesh model.

[0041] In one possible implementation, the three-dimensional model reconstruction method further includes adjusting the coordinates of each point in the three-dimensional mesh model according to the type of each target object and the corresponding mapping range in the three-dimensional mesh model to obtain an optimized three-dimensional mesh model.

[0042] In an embodiment of the present application, based on the same coordinate point adjustment method, the coordinates of each point in the three-dimensional mesh model after three-dimensional reconstruction are adjusted again, thereby improving the problem of large deviations between the reconstructed model and the actual model caused by incorrect connection of triangular faces in the reconstructed mesh model, and further increasing the accuracy of three-dimensional reconstruction.

[0043] In one possible implementation, the three-dimensional model reconstruction method also includes extracting image texture from the two-dimensional color image set, mapping the image texture to the three-dimensional mesh model or the optimized three-dimensional mesh model according to the corresponding rotation matrix and translation vector, and obtaining a textured three-dimensional mesh model.

[0044] In an embodiment of the present application, after obtaining the three-dimensional mesh model, image texture is further extracted from the two-dimensional color image set, and the image texture is mapped to the three-dimensional mesh model to obtain a textured three-dimensional mesh model, thereby further improving the accuracy of three-dimensional reconstruction.

[0045] In a second aspect, the present application provides a three-dimensional model reconstruction system for a local dark and weak texture scene, including an acquisition module, a target detection module, a feature detection module, a target mapping module, an adjustment module, and a network reconstruction module;

[0046] The acquisition module is used to acquire an original two-dimensional color image set and perform brightness and chromaticity normalization adjustment on the original two-dimensional color image set to obtain a corresponding two-dimensional color image set;

[0047] The target detection module is used to input the two-dimensional color image set into a pre-trained neural network structure model, so that the neural network structure model can identify a number of target objects in the two-dimensional color image set and obtain the corresponding position and type of each target object in each frame image;

[0048] The feature detection module is used to perform feature point detection on the two-dimensional color image set, determine the rotation matrix and translation vector corresponding to each frame image, and construct an initial three-dimensional point cloud model corresponding to the two-dimensional color image set;

[0049] The target mapping module is used to map the corresponding position of each target object in each frame image to the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image, and then determine the corresponding mapping range of each target object in the initial three-dimensional point cloud model;

[0050] The adjustment module is used to adjust the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model;

[0051] The network reconstruction module is used to reconstruct the mesh of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model.

[0052] In one possible implementation, the three-dimensional model reconstruction system also includes a texture mapping module, which is used to extract image texture from the two-dimensional color image set, map the image texture to the three-dimensional mesh model according to the corresponding rotation matrix and translation vector, and obtain a textured three-dimensional mesh model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 : A flow chart of a method for reconstructing a three-dimensional model for a scene with local dim texture provided in an embodiment of the present application.

[0054] Figure 2 : A schematic diagram of the process of constructing an initial three-dimensional point cloud model in a three-dimensional model reconstruction method for a local dark and weak texture scene provided in an embodiment of the present application.

[0055] Figure 3 : A schematic diagram of a flow chart for determining the mapping range of each target object in a three-dimensional model reconstruction method for a local dark and weak texture scene provided in an embodiment of the present application.

[0056] Figure 4 : A structural schematic diagram of a three-dimensional model reconstruction system for a local dark and weak texture scene provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] It should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments and do not serve to limit the order in which the steps are executed. In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0059] Embodiment 1:

[0060] like Figure 1 As shown, the first embodiment provides a 3D model reconstruction method for a local dark and weak texture scene, including steps S1-S6:

[0061] Step S1, obtaining an original two-dimensional color image set and performing brightness and chromaticity normalization adjustment on the original two-dimensional color image set to obtain a corresponding two-dimensional color image set;

[0062] Step S2, inputting the two-dimensional color image set into a pre-trained neural network structure model, so that the neural network structure model recognizes a number of target objects in the two-dimensional color image set, and obtains the corresponding position and type of each target object in each frame image;

[0063] Step S3, performing feature point detection on the two-dimensional color image set, determining the rotation matrix and translation vector corresponding to each frame image, and constructing an initial three-dimensional point cloud model corresponding to the two-dimensional color image set;

[0064] Step S4, mapping the corresponding position of each target object in each frame image to the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image, and then determining the corresponding mapping range of each target object in the initial three-dimensional point cloud model;

[0065] Step S5, adjusting the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model;

[0066] Step S6: reconstruct the mesh of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model.

[0067] The embodiment of the present application provides a three-dimensional model reconstruction method for a local dark and weak texture scene. First, the brightness difference of images taken at adjacent angles of the same scene is eliminated by adjusting the brightness and chromaticity normalization of the original two-dimensional color image set, which can not only improve the detection amount of feature points and reduce the failure rate of reconstruction, but also, different from the traditional texture fusion technology, avoids the phenomenon of scene breakage caused by the inconsistent brightness caused by the use of adjacent images by adjacent grids after texture mapping of the reconstructed grid model, and solves the problem of unnatural texture transition. Further, the two-dimensional color image set is detected by the neural network structure model to identify several target objects in the image, which provides data support for the subsequent three-dimensional grid reconstruction. The target detection by the neural network structure model also solves the problem of partial missing or even reconstruction failure of the reconstruction model caused by unclear images, gray, weak features, etc. to a certain extent. Then, according to each rotation matrix and translation vector, the corresponding mapping range of each target object in the initial three-dimensional point cloud model is determined, and then the point coordinates within the mapping range are adjusted according to the type of the target object, the point cloud position is adjusted, and the problem of laser point offset caused by reflective materials is solved, and the success rate and accuracy of reconstruction are further increased. Finally, the 3D point cloud model is meshed and reconstructed to obtain a corresponding 3D mesh model, realizing a 3D reconstruction method combining traditional visual geometry and neural network technology, thereby improving the success rate and accuracy of 3D model reconstruction for local dark and weak texture scenes.

[0068] In a preferred embodiment, since the image set may be too bright, too dark, or biased towards a certain color system due to various reasons during the two-dimensional color image data acquisition process, each pixel of each frame of the image is normalized in step S1 to obtain a two-dimensional color image set with consistent brightness and chromaticity, specifically:

[0069] There are N frames of images, each frame of the original color image {I i , i=1,…,N} width is w, height is h, and the values ​​of the three basic colors of red, green and blue in the three channels of each pixel (x, y) are:

[0070]

[0071] Calculate the mean for each channel of each frame of the original image:

[0072]

[0073] Normalization parameters:

[0074]

[0075] Create a normalized lookup table:

[0076]

[0077] Where c represents the value of one of the three RGB chromaticities in a single pixel, Table c Represents the normalized chromaticity value.

[0078] Finally, each pixel of each frame of the original image is normalized to obtain the corresponding two-dimensional color image set: when

[0079] Compared with the original image set, the normalized image set can eliminate the local overly bright or overly dark areas and highlight the scene features.

[0080] In a possible implementation, in step S2, the neural network structure model is constructed based on the neural network structure model of Yolov8n and obtained through training with a number of training data, including:

[0081] Determining a plurality of target types to be identified based on the original two-dimensional color image set;

[0082] Acquiring a plurality of two-dimensional color image data containing the target type;

[0083] Performing brightness and chromaticity normalization adjustment on each of the two-dimensional color image data to obtain a corresponding training data set;

[0084] Construct an initial neural network structure model based on Yolov8n;

[0085] The initial neural network structure model is trained to detect network weight coefficients using the training data set, and each weight coefficient in the initial neural network structure model is determined to obtain the neural network structure model.

[0086] In a preferred embodiment, 7 common targets are selected as target types to be identified, including cars, bicycles, electric vehicles, flat walls, windows, doors, stairs and other target types. Then, a public data set containing these 7 types of targets and a self-collected data set are combined as a training data set, and the training data set is normalized for brightness and chromaticity. The neural network structure model of Yolov8n is used to train the weight coefficient of the detection network, obtain the model weight, and then obtain the neural network structure model, wherein each training parameter in the model takes the default value.

[0087] An embodiment of the present application provides a method for training a neural network structure model. First, based on the content of an original two-dimensional color image set, several target types to be identified are determined. Then, based on the determined target types, several two-dimensional color image data are collected and brightness and chromaticity normalization adjustments are performed as corresponding training data sets. Brightness and chromaticity normalization adjustments are performed on the two-dimensional color image data, so that the initial neural network structure model adapts to the preprocessed image data in advance during the training process, thereby improving the accuracy of the model in recognizing the content of the two-dimensional color image set.

[0088] In a preferred embodiment, the neural network structure model in step S2 identifies a plurality of target objects in the two-dimensional color image set, and obtains the corresponding position and type of each of the target objects in each frame image, specifically:

[0089] Set the confidence threshold to 0.8, and use the trained neural network structure model to detect the two-dimensional color image frame by frame to obtain the position of the jth target appearing in the i-th frame image. in is the confidence of the current target, and It is the offset of the upper left corner of the current target rectangle from the left and top sides of the image. and is the width and height of the rectangular box where the current target is located. Indicates the type of the current target.

[0090] Further, in step S3, feature point detection is performed on the two-dimensional color image set to determine the rotation matrix and translation vector corresponding to each frame image, and an initial three-dimensional point cloud model corresponding to the two-dimensional color image set is constructed, such as Figure 2 As shown, steps S301-S305 are included:

[0091] Step S301, obtaining parameters of a camera for shooting the two-dimensional color image set;

[0092] Step S302, performing scale-invariant feature point detection on the two-dimensional color image set frame by frame to obtain a feature point set corresponding to each frame image;

[0093] Step S303, performing feature matching on each feature point of each frame image with each feature point of the next frame image, and calculating and obtaining the rotation matrix and translation vector corresponding to each frame image according to the feature matching result;

[0094] Step S304, calculating and obtaining the three-dimensional coordinate point set corresponding to the feature point set of each frame image according to the camera parameters and the feature point set, rotation matrix and translation vector corresponding to each frame image;

[0095] Step S305: constructing and obtaining the initial three-dimensional point cloud model according to each of the three-dimensional coordinate point sets.

[0096] In a preferred embodiment, the specific implementation process of steps S301-S305 is:

[0097] The scale invariant feature (SIFT) is used as the image feature to transform the two-dimensional color image set. Perform feature point detection on each frame image in the image to obtain the feature point set F corresponding to each frame image i ={x n ,y n ,D},n=1,…,N F , indicating that the frame has a total of N F features, and the image coordinates of each feature are (x n ,y n ), the description of the feature is D.

[0098] Considering that the input color image set is mainly composed of continuous video frames, and there is a correlation between the previous and next frames, the scheme of comparing and matching feature points of the previous and next frames is adopted, that is, for the image Each feature description point in Calculate adjacent images Each feature description point in and The distance and select the nearest k = 1 feature description points as feature description points In the image The matching feature description points in

[0099] Remember adjacent images arrive During the shooting process, the relative transformation matrix of the camera is R i,j , the relative translation vector is t i,j , then for the matching pair Match i,j Any pair of matching feature description points Both Write the feature point set in matrix form:

[0100]

[0101] Then there is F i =[R i,j ,t i,j ]F j , the relative transformation matrix R can be obtained based on the matching pairs of adjacent images i,jand the relative translation vector t i,j :

[0102] [R i,j ,t i,j ]=F i (F j ) -1

[0103] The focal length of the camera is f = (f x ,f y ), the lens distortion coefficient is c=(c x ,c y ), then the intrinsic parameter matrix of the camera is:

[0104]

[0105] Record feature description points The three-dimensional coordinates mapped to the scene are Then there is Then the image Characteristic description points Mapped 3D coordinate point set It is the 3D point cloud model of the part of the scene captured by the i-th frame image. The 3D coordinate point set mapped by all frame images together constitutes the 3D point cloud model of the entire scene.

[0106] The embodiment of the present application provides a method for detecting feature points of a two-dimensional color image set, and performs scale-invariant feature point detection on the two-dimensional color image set frame by frame. Considering that the two-dimensional color image set is mainly composed of continuous video frames, and there is a correlation between the previous and next frames, the embodiment of the present application adopts a solution of comparing and matching feature points of the previous and next frames, and performs feature matching on the feature points of the detected adjacent frame images in pairs, thereby obtaining the rotation matrix and translation vector corresponding to each frame image, as the basis for mapping from two dimensions to three dimensions. Then, the three-dimensional coordinate point set of each frame image is calculated according to the camera parameters and the feature point set, rotation matrix and translation vector corresponding to each frame image, and then the initial three-dimensional point cloud model is constructed according to each three-dimensional coordinate point set, and the initial three-dimensional point cloud model contains all the feature points in the two-dimensional color image set after being mapped to the three-dimensional space, providing a data basis for subsequent three-dimensional reconstruction.

[0107] Further, in step S4, the corresponding position of each target object in each frame image is mapped to the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image, and then the corresponding mapping range of each target object in the initial three-dimensional point cloud model is determined, such as Figure 3 As shown, steps S401-S403 are included:

[0108] Step S401, calculating the three-dimensional coordinates of the corresponding positions of each target object in each frame image after mapping according to the rotation matrix and translation vector corresponding to each frame image, and then constructing the initial three-dimensional mapping point set corresponding to each target object;

[0109] Step S402: coordinate screening of the initial three-dimensional mapping point set corresponding to each target object according to a preset screening rule to obtain a three-dimensional mapping point set corresponding to each target object;

[0110] Step S403: determining a mapping range corresponding to each of the target objects in the initial three-dimensional point cloud model according to each of the three-dimensional mapping point sets.

[0111] In a preferred embodiment, the specific implementation process of steps S401-S403 is:

[0112] According to the relative transformation matrix R corresponding to each frame image i,j and the relative translation vector t i,j By mapping the target rectangular frame detected in each image to the three-dimensional space, the position of the target rectangular frame contained in the three-dimensional point cloud model P can be obtained. j}={p j ,x j ,y j ,z j ,X j ,Y j ,Z j ,c j}, where p j is the confidence of the current target, x j ,y j ,z j is the 3D coordinate of the center of the target cuboid, X j ,Y j ,Z j is the length of the target cuboid on three axes, c j Indicates the type of the current target.

[0113] The specific operation is: record the image coordinate information of the jth target in the i-th frame image as Then the coordinates of the upper left corner of the rectangular box of the target in the image are The coordinates of the lower right corner are The corresponding three-dimensional coordinates in space are:

[0114] P LeftTop =K[R 0,i ,t 0,i ]p LeftTop

[0115] P RightBottom =K[R0,i ,t 0,i ]p RightBottom

[0116] Note that the target appears from the i-th frame to the i+s-th frame, and the mapping point set of the target in the three-dimensional space is obtained:

[0117]

[0118] The three-dimensional cuboid frame of the target is determined by the two corner points closest and farthest from the origin according to the Euclidean distance, which are:

[0119]

[0120] In the embodiment of the present application, the three-dimensional coordinates of the corresponding positions of each target object in each frame image after mapping are calculated by rotation matrix and translation vector, that is, the position range of each target object in the two-dimensional image is mapped to the three-dimensional space in a point-to-point form, and the initial three-dimensional mapping point set corresponding to each target object is constructed. Similarly, considering that the two-dimensional color image set is mainly continuous video frames, the same object will appear repeatedly in the images of different frames, so multiple position ranges will be repeatedly mapped to the three-dimensional space. Therefore, it is necessary to screen each initial three-dimensional mapping point set in the future to determine the three-dimensional mapping point set after screening for each target object. Finally, the corresponding mapping range of each target object in the initial three-dimensional point cloud model can be determined according to each three-dimensional mapping point set, and the points within a certain mapping range in the initial three-dimensional point cloud model are the points formed after the corresponding target object is mapped from the two-dimensional space to the three-dimensional space.

[0121] In a possible implementation, in step S5, adjusting the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model includes:

[0122] According to the type of each target object, extracting corresponding three-dimensional mesh object models from a plurality of preset three-dimensional mesh object models;

[0123] For any first target object among the target objects, adjusting the coordinates of each point in the corresponding first mapping range in the initial three-dimensional point cloud model according to each vertex position and triangle face position in the first three-dimensional mesh object model corresponding to the first target object, so as to make the coordinates of each point in the first mapping range more closely fit the first three-dimensional mesh object model;

[0124] After the coordinate adjustment is completed, the three-dimensional point cloud model is obtained.

[0125] In a preferred embodiment, the specific implementation process of the above method is:

[0126] According to the types of the preset 7 types of target objects, 80-100 3D mesh models are extracted from the public data set of 3D network models to form the several 3D mesh object models. Each target type c j (j=1,…,7) corresponds to a type of 3D mesh object model

[0127] The coordinates of each point in the mapping range corresponding to each target object are detected and adjusted in sequence. Taking any first target object among the target objects as an example, according to the type c of the first target object, j , randomly extract a first three-dimensional mesh object model from multiple three-dimensional mesh object models of the corresponding type where v c is the vertex set of the first three-dimensional mesh model, f c is the triangular face set of the first three-dimensional mesh object model. The three-dimensional point cloud set within the mapping range of the first target object is According to the first 3D mesh model A 3D point cloud set within the mapping range of the first target object Deform and fine-tune the position of the point concentration point to make it fit as closely as possible.

[0128] The specific operation is: first and Do ICP transformation so that and {P cj} are in the same direction, that is, c All vertices v in c ,remember The closest point to it is P c , minimize v c and P c Euclidean distance between pairs of points:

[0129]

[0130] Then calculate separately and {P cj The point density of the} is the number of points within a unit distance ρ with each point as the center P .right and The part with low point density is used for mesh or point cloud densification. The deformation factor α is taken as a two-dimensional Gaussian function Calculate the initial deformation point P c′ =αv c +(1-α)P c , the final deformation point Where δ(P c ′ ) is point P c ′ 's neighborhood.

[0131] In the embodiment of the present application, the coordinates of each point in each mapping range in the initial three-dimensional point cloud model are adjusted according to the type of target object corresponding to each mapping range. First, it is necessary to determine each three-dimensional mesh object model of the same type as each target object for reference. Then, according to each three-dimensional mesh object model, the coordinates of each point in each mapping range are adjusted respectively, so that the coordinates of each point are more closely matched to the corresponding three-dimensional mesh object model, which solves the problem of laser point offset caused by reflective materials and further increases the success rate and accuracy of three-dimensional reconstruction.

[0132] Furthermore, in step S6, the meshing of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model includes:

[0133] Triangulating and segmenting the three-dimensional point cloud model, and converting the three-dimensional point cloud model into a set consisting of a plurality of vertices, tetrahedrons and triangular faces;

[0134] Calculate the weight of each vertex according to the preset formula;

[0135] According to each of the weights and the orientation of each of the triangular faces, the set is segmented using maximum flow minimum cut to generate the three-dimensional mesh model.

[0136] In a preferred embodiment, the specific implementation process of the above method is:

[0137] First, perform Delaunay triangulation on the 3D point cloud model of the entire scene and transform the point cloud into a set {P} = {v, c, f} containing vertex v, tetrahedron c and triangle face f. For each vertex v, calculate the visibility weight Where d is the sum of the Euclidean distances between the tetrahedron faces and vertex v that the ray from the camera to vertex v passes through, and σ = 1.0 is the weight attenuation coefficient. Finally, according to the weight and the orientation of each triangle, the maximum flow minimum cut is used for segmentation, and the segmentation boundary is the generated three-dimensional mesh model M 0 .

[0138] In one possible implementation, the three-dimensional model reconstruction method further includes adjusting the coordinates of each point in the three-dimensional mesh model according to the type of each target object and the corresponding mapping range in the three-dimensional mesh model to obtain an optimized three-dimensional mesh model.

[0139] In an embodiment of the present application, based on the same coordinate point adjustment method, the coordinates of each point in the three-dimensional mesh model after three-dimensional reconstruction are adjusted again, thereby improving the problem of large deviations between the reconstructed model and the actual model caused by incorrect connection of triangular faces in the reconstructed mesh model, and further increasing the accuracy of three-dimensional reconstruction.

[0140] In one possible implementation, the three-dimensional model reconstruction method also includes extracting image texture from the two-dimensional color image set, mapping the image texture to the three-dimensional mesh model or the optimized three-dimensional mesh model according to the corresponding rotation matrix and translation vector, and obtaining a textured three-dimensional mesh model.

[0141] In a preferred embodiment, the specific implementation process of the above method is:

[0142] Taking the optimization of the three-dimensional mesh model as an example, the optimized three-dimensional mesh model is denoted as M 1 = {v,c,f,e}, where v,c,f,e represent vertices, tetrahedrons, triangles, and edges in the model respectively. For each triangle f consisting of three vertices, 1 ,v 2 ,v 3}, let its normal vector be n, projected onto image I i The triangle on The center of gravity is Traverse the rotation matrix R and translation vector t corresponding to the two-dimensional color image set and each frame image, record the camera position as C, select the corresponding image under the optimal viewing angle, and extract the image texture of the image, that is, Where γ = 0.5 is the orientation and position bias coefficient, and then the image texture is mapped to the three-dimensional mesh model.

[0143] In the image texture mapping process, for the neighborhood where the triangles intersect each other, that is, the mesh model M 1 All connected edges {e} of , use matching blocks to repair unnatural texture transitions. That is, take the repair kernel K = E 3 , connect the edge e 12 Connected adjacent triangles f 1 and f 2 Flattened into a plane frame π, for any containing e 12 The box of the repair kernel size at the upper point is randomly selected to correspond to the two-dimensional image I 1 and I 2 The boxes of the same size in the , and the matching degree is calculated, and the best matching box is selected for iterative repair.

[0144] Finally, we get a 3D mesh model with texture {M1 ,T}.

[0145] In the embodiment of the present application, after obtaining the three-dimensional mesh model, the image texture is further extracted from the two-dimensional color image set, and the image texture is mapped to the three-dimensional mesh model to obtain a textured three-dimensional mesh model, thereby further improving the accuracy of the three-dimensional reconstruction. In addition, in the texture mapping process, the matching block-based texture mapping solution provided in the embodiment of the present application eliminates the unnatural connection or even fault phenomenon in the texture mapping process of the reconstructed mesh model.

[0146] Embodiment 2:

[0147] like Figure 4 As shown, the second embodiment provides a 3D model reconstruction system for a local dark and weak texture scene, including an acquisition module 10, a target detection module 20, a feature detection module 30, a target mapping module 40, an adjustment module 50 and a network reconstruction module 60;

[0148] The acquisition module 10 is used to acquire an original two-dimensional color image set and perform brightness and chromaticity normalization adjustment on the original two-dimensional color image set to obtain a corresponding two-dimensional color image set;

[0149] The target detection module 20 is used to input the two-dimensional color image set into a pre-trained neural network structure model, so that the neural network structure model can identify a number of target objects in the two-dimensional color image set and obtain the corresponding position and type of each target object in each frame image;

[0150] The feature detection module 30 is used to perform feature point detection on the two-dimensional color image set, determine the rotation matrix and translation vector corresponding to each frame image, and construct an initial three-dimensional point cloud model corresponding to the two-dimensional color image set;

[0151] The target mapping module 40 is used to map the corresponding position of each target object in each frame image to the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image, and then determine the corresponding mapping range of each target object in the initial three-dimensional point cloud model;

[0152] The adjustment module 50 is used to adjust the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model;

[0153] The network reconstruction module 60 is used to reconstruct the mesh of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model.

[0154] In one possible implementation, the three-dimensional model reconstruction system also includes a texture mapping module, which is used to extract image texture from the two-dimensional color image set, map the image texture to the three-dimensional mesh model according to the corresponding rotation matrix and translation vector, and obtain a textured three-dimensional mesh model.

[0155] The embodiment of the present application provides a three-dimensional model reconstruction system for a local dark and weak texture scene. First, the brightness difference of images taken at adjacent angles of the same scene is eliminated by adjusting the brightness and chromaticity normalization of the original two-dimensional color image set, which can not only improve the detection amount of feature points and reduce the failure rate of reconstruction, but also, different from the traditional texture fusion technology, avoid the phenomenon of scene breakage caused by the inconsistent brightness caused by the use of adjacent images by adjacent grids after texture mapping of the reconstructed grid model, and solve the problem of unnatural texture transition. Further, the two-dimensional color image set is detected by the neural network structure model to identify several target objects in the image, which provides data support for the subsequent three-dimensional grid reconstruction. The target detection by the neural network structure model also solves the problem of partial missing or even reconstruction failure of the reconstruction model caused by unclear images, gray, weak features, etc. to a certain extent. Then, according to each rotation matrix and translation vector, the corresponding mapping range of each target object in the initial three-dimensional point cloud model is determined, and then the point coordinates within the mapping range are adjusted according to the type of the target object, the point cloud position is adjusted, and the problem of laser point offset caused by reflective materials is solved, and the success rate and accuracy of reconstruction are further increased. Finally, the 3D point cloud model is meshed and reconstructed to obtain a corresponding 3D mesh model, realizing a 3D reconstruction method combining traditional visual geometry and neural network technology, thereby improving the success rate and accuracy of 3D model reconstruction for local dark and weak texture scenes.

[0156] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.

[0157] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A 3D model reconstruction method for a local dark and weak texture scene, characterized in that: include: Acquire an original two-dimensional color image set and perform brightness and chromaticity normalization adjustment on the original two-dimensional color image set to obtain a corresponding two-dimensional color image set; Inputting the two-dimensional color image set into a pre-trained neural network structure model, so that the neural network structure model recognizes a number of target objects in the two-dimensional color image set, and obtains the corresponding position and type of each target object in each frame image; Performing feature point detection on the two-dimensional color image set, determining the rotation matrix and translation vector corresponding to each frame image, and constructing an initial three-dimensional point cloud model corresponding to the two-dimensional color image set; According to the rotation matrix and translation vector corresponding to each frame image, the corresponding position of each target object in each frame image is mapped to the initial three-dimensional point cloud model, thereby determining the corresponding mapping range of each target object in the initial three-dimensional point cloud model; According to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model, the coordinates of each point in the initial three-dimensional point cloud model are adjusted to obtain a three-dimensional point cloud model; The three-dimensional point cloud model is meshed to obtain a corresponding three-dimensional mesh model.

2. A 3D model reconstruction method for a local dark texture scene according to claim 1, characterized in that: The neural network structure model is constructed based on the neural network structure model of Yolov8n and is obtained through training with a number of training data, including: Determining a plurality of target types to be identified based on the original two-dimensional color image set; Acquiring a plurality of two-dimensional color image data containing the target type; Performing brightness and chromaticity normalization adjustment on each of the two-dimensional color image data to obtain a corresponding training data set; Construct an initial neural network structure model based on Yolov8n; The initial neural network structure model is trained to detect network weight coefficients using the training data set, and each weight coefficient in the initial neural network structure model is determined to obtain the neural network structure model.

3. The three-dimensional model reconstruction method for a local dark and weak texture scene according to claim 1, characterized in that: The method of performing feature point detection on the two-dimensional color image set, determining the rotation matrix and translation vector corresponding to each frame image, and constructing an initial three-dimensional point cloud model corresponding to the two-dimensional color image set includes: Obtaining parameters of a camera used to capture the two-dimensional color image set; Performing scale-invariant feature point detection on the two-dimensional color image set frame by frame to obtain a feature point set corresponding to each frame image; Perform feature matching on each feature point of each frame image with each feature point of the next frame image, and calculate the rotation matrix and translation vector corresponding to each frame image according to the feature matching results; According to the camera parameters and the feature point set corresponding to each frame image, the rotation matrix and the translation vector, respectively calculate and obtain the three-dimensional coordinate point set corresponding to the feature point set of each frame image; The initial three-dimensional point cloud model is constructed based on each of the three-dimensional coordinate point sets.

4. A 3D model reconstruction method for a local dark texture scene as claimed in claim 1, characterized in that: The mapping of the corresponding position of each target object in each frame image to the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image, and then determining the corresponding mapping range of each target object in the initial three-dimensional point cloud model, includes: According to the rotation matrix and translation vector corresponding to each frame image, the three-dimensional coordinates of the corresponding position of each target object in each frame image after mapping are calculated, and then the initial three-dimensional mapping point set corresponding to each target object is constructed; According to a preset screening rule, coordinate screening is performed on the initial three-dimensional mapping point set corresponding to each of the target objects to obtain the three-dimensional mapping point set corresponding to each of the target objects; According to each of the three-dimensional mapping point sets, a mapping range corresponding to each of the target objects in the initial three-dimensional point cloud model is determined.

5. The three-dimensional model reconstruction method for a local dark and weak texture scene according to claim 1, characterized in that: The step of adjusting the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model includes: According to the type of each target object, extracting corresponding three-dimensional mesh object models from a plurality of preset three-dimensional mesh object models; For any first target object among the target objects, adjusting the coordinates of each point in the corresponding first mapping range in the initial three-dimensional point cloud model according to each vertex position and triangle face position in the first three-dimensional mesh object model corresponding to the first target object, so as to make the coordinates of each point in the first mapping range more closely fit the first three-dimensional mesh object model; After the coordinate adjustment is completed, the three-dimensional point cloud model is obtained.

6. A 3D model reconstruction method for a local dark and weak texture scene as claimed in claim 1, characterized in that: The step of reconstructing the mesh of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model includes: Triangulating and segmenting the three-dimensional point cloud model, and converting the three-dimensional point cloud model into a set consisting of a plurality of vertices, tetrahedrons and triangular faces; Calculate the weight of each vertex according to the preset formula; According to each of the weights and the orientation of each of the triangular faces, the set is segmented using maximum flow minimum cut to generate the three-dimensional mesh model.

7. The method for reconstructing a three-dimensional model for a scene with local dark and weak textures according to claim 1, characterized in that: The three-dimensional model reconstruction method further includes adjusting the coordinates of each point in the three-dimensional grid model according to the type of each target object and the corresponding mapping range in the three-dimensional grid model to obtain an optimized three-dimensional grid model.

8. A 3D model reconstruction method for a local dark and weak texture scene as claimed in claim 7, characterized in that: The three-dimensional model reconstruction method also includes extracting image texture from the two-dimensional color image set, mapping the image texture to the three-dimensional mesh model or the optimized three-dimensional mesh model according to the corresponding rotation matrix and translation vector, and obtaining a textured three-dimensional mesh model.

9. A 3D model reconstruction system for a local dark and weak texture scene, characterized in that: It includes an acquisition module, a target detection module, a feature detection module, a target mapping module, an adjustment module and a network reconstruction module; The acquisition module is used to acquire an original two-dimensional color image set and perform brightness and chromaticity normalization adjustment on the original two-dimensional color image set to obtain a corresponding two-dimensional color image set; The target detection module is used to input the two-dimensional color image set into a pre-trained neural network structure model, so that the neural network structure model can identify a number of target objects in the two-dimensional color image set and obtain the corresponding position and type of each target object in each frame image; The feature detection module is used to perform feature point detection on the two-dimensional color image set, determine the rotation matrix and translation vector corresponding to each frame image, and construct an initial three-dimensional point cloud model corresponding to the two-dimensional color image set; The target mapping module is used to map the corresponding position of each target object in each frame image to the initial three-dimensional point cloud model according to the rotation matrix and translation vector corresponding to each frame image, and then determine the corresponding mapping range of each target object in the initial three-dimensional point cloud model; The adjustment module is used to adjust the coordinates of each point in the initial three-dimensional point cloud model according to the type of each target object and the corresponding mapping range in the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model; The network reconstruction module is used to reconstruct the mesh of the three-dimensional point cloud model to obtain a corresponding three-dimensional mesh model.

10. The 3D model reconstruction system for a local dark and weak texture scene according to claim 9, characterized in that: The three-dimensional model reconstruction system also includes a texture mapping module, which is used to extract image textures from the two-dimensional color image set, map the image textures to the three-dimensional mesh model according to the corresponding rotation matrix and translation vector, and obtain a three-dimensional mesh model with textures.

Images