Scene construction method and device, electronic equipment and storage medium

Abstract

The invention discloses a scene construction method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: arranging a preset light source and a camera at the same coordinate point, and obtaining the position of the preset light source through determining the pose of the camera in the moving process. The technical problem that the illumination direction of the reconstructed target surface in the moving state cannot be estimated can be solved, and the technical effect of improving the application scene of the photometric stereo method is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a scene construction method and device, electronic equipment and storage medium. Background Art

[0002] Photometric stereo method uses a camera to capture the changes in light reflection from different parts of an object by adding additional lighting, calculates the fixed normal vector of the object surface, and finally uses integration and other methods to obtain the surface information of the object. In the process of the camera acquiring data, fast movement of the camera will cause the position of the camera and light source relative to the reconstructed target to change, making it impossible to estimate the light direction of the reconstructed target surface. Summary of the invention

[0003] The present application provides a scene construction method and device, electronic device and storage medium, so as to at least solve the problem that the illumination direction of the reconstructed target surface cannot be estimated in the related art.

[0004] This application provides a method for constructing a scene, including: The normal vector of each point in the 3D point cloud is calculated based on the camera pose and the 3D point cloud of the reconstructed target; wherein the camera pose and the 3D point cloud are calculated based on the preset light source at the same coordinate point and the data collected by the camera; Perform triangulation based on the three-dimensional point cloud and the normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point; Determine a first target iteration point according to a photometric error of at least one iteration point; The first target iteration point is placed in the three-dimensional point cloud, and the three-dimensional point cloud is iteratively triangulated until the scale of the three-dimensional point cloud reaches a preset scale value.

[0005] The present application also provides a scene construction device, including: A first calculation unit is used to calculate the normal vector of each point in the three-dimensional point cloud based on the camera posture and the three-dimensional point cloud of the reconstructed target; wherein the camera posture and the three-dimensional point cloud are calculated based on the preset light source at the same coordinate point and the data collected by the camera; A determination unit, configured to perform triangulation based on the three-dimensional point cloud and the normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point; A second calculation unit, used to determine a first target iteration point according to a photometric error of at least one iteration point; The iterative calculation unit is used to place the first target iteration point in the three-dimensional point cloud and iteratively triangulate the three-dimensional point cloud until the scale of the three-dimensional point cloud reaches a preset scale value.

[0006] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the construction method of any of the above scenarios when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the construction method of any of the above scenarios are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of the construction method of any of the above scenarios when the computer program is executed by a processor.

[0009] Through the present application, since the preset light source and the camera are arranged at the same coordinate point, the position of the preset light source can be known by determining the posture of the camera during movement. Therefore, the technical problem of being unable to estimate the illumination direction of the reconstructed target surface in a moving state can be solved, thereby achieving the technical effect of improving the applicable scenarios of the photometric stereo method. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work: Figure 1 A schematic diagram of a flow chart of a method for constructing a scenario provided in an embodiment of the present application; Figure 2 A schematic diagram of the arrangement position of a camera and a preset light source provided in an embodiment of the present application; Figure 3 A flowchart of another method for constructing a scenario provided in an embodiment of the present application; Figure 4 A schematic diagram of a lighting model provided in an embodiment of the present application; Figure 5 A flowchart of another method for constructing a scenario provided in an embodiment of the present application; Figure 6 A schematic diagram of an iterative calculation process provided in an embodiment of the present application; Figure 7 A triangulation schematic diagram provided in an embodiment of the present application; Figure 8 A flowchart of another method for constructing a scenario provided in an embodiment of the present application; Fig. 9 A schematic diagram of the structure of a scene construction device provided in an embodiment of the present application; Fig.10 A schematic diagram of the structure of a construction device for another scenario provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0013] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0014] Figure 1 The following is a flow chart of a method for constructing a scene provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps: Step 101, calculate the normal vector of each point in the three-dimensional point cloud based on the camera posture and the three-dimensional point cloud of the reconstructed target; wherein the camera posture and the three-dimensional point cloud are calculated based on the preset light source at the same coordinate point and the data collected by the camera.

[0015] See also Figure 2 , Figure 2 A schematic diagram of the arrangement position of a camera and a preset light source provided in an embodiment of the present application, such as Figure 2 As shown in the figure, the camera and the preset light are set close to each other, resulting in the distance between the camera and the light source being much smaller than the distance between the camera and the reconstructed target. The camera and the preset light can be considered to be arranged at the same coordinate point, and the position of the preset light can be equated with the position of the camera. After obtaining the camera's pose through the motion recovery structure method, the constantly changing position of the preset light source during the movement can be known.

[0016] Structure from Motion (SfM) is a technique for recovering the structure of a 3D scene from multi-view 2D images. By analyzing the matching relationship between feature points between images and combining the camera motion information, the 3D point cloud of the scene and the camera's position and posture are reconstructed. The core process is as follows: Extract feature point information from a two-dimensional image sequence, that is, find points in the image that are easily distinguishable to facilitate subsequent positioning.

[0017] The feature points in the image sequence are matched pairwise to obtain the projections of the points in the three-dimensional scene at different shooting angles.

[0018] Select two initial images and use the two-dimensional to two-dimensional (2D-2D) method to solve the camera pose.

[0019] According to the solved pose and matched feature points, triangulation is used to solve the three-dimensional coordinates of the feature points in the three-dimensional scene.

[0020] Then select a new image and use the 3D to 2D (3D-2D) method to solve the camera pose.

[0021] The feature points in the new image are added to the new 3D scene point cloud through triangulation.

[0022] The camera pose and 3D scene point cloud are optimized using bundle adjustment to avoid cascading errors, and the above steps are repeated until no new images are available for processing.

[0023] See also Figure 3 , Figure 3 A flow chart of another method for constructing a scenario provided in an embodiment of the present application is shown as follows: Figure 3 As shown in S3, after knowing the 3D point cloud data, camera pose and preset light position, the 3D point cloud (i.e. Figure 3 The normal vector of each point in the sparse point cloud) is The calculation formula is:

[0024] in, is the illumination intensity of the point, S is the coordinate of the point in the world coordinate system, is the position of the kth camera and preset light source.

[0025] Step 102 : triangulate the three-dimensional point cloud and the normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point.

[0026] Triangulation converts 3D point cloud data into a triangular mesh model, which can more intuitively represent the surface structure of an object. After completing the acquisition and preliminary processing of the 3D point cloud, a large amount of scattered spatial point data is obtained. Although these point data can reflect the shape of the object to a certain extent, they lack continuity and structure. Based on the triangulation process, the points in the point cloud are connected through triangular facets to form a continuous triangular mesh model.

[0027] In some embodiments, when performing triangulation, an algorithm is selected from the triangulation algorithms, such as Delaunay triangulation, incremental method (ncremental) and other algorithms. Specifically, it can be determined according to actual needs, such as: when the generation quality requirement is high, the Delaunay triangulation algorithm can be selected; when the triangulation object is a simple polygon, the ear-cut triangulation algorithm can be selected. It should be noted that this description is only an exemplary description, not a specific limitation on a specific triangulation algorithm, and the embodiments of the present application do not limit this.

[0028] Step 103: determine a first target iteration point according to the photometric error of at least one iteration point.

[0029] Please continue reading Figure 3 ,like Figure 3 In S4, it is recorded that triangulation is performed based on the normal vector of each point in the point cloud data. The purpose of triangulation is to find a set of optimal triangular facets in the point cloud so that the triangular facets can reflect the real surface of the object as much as possible. The coordinates of the centroid of the triangular facet are determined by the coordinate values ​​of the three vertices of the triangular facet, and the iteration point is determined based on the centroid. It should be noted that each triangular facet corresponds to an iteration point, and the photometric error of the iteration point is calculated, and the iteration point with the smallest photometric error is determined as the first target iteration point.

[0030] Repeat the above steps until the photometric error of the entire model is reduced to an acceptable range or the preset number of iterations is reached.

[0031] Photometric error refers to the error caused by inaccurate measurement of light intensity, color, direction or the optical properties of the imaging system lens, sensor non-uniformity and changes in ambient light in optical measurement or image processing.

[0032] Step 104 , placing the first target iteration point in the three-dimensional point cloud, and iteratively triangulating the three-dimensional point cloud until the scale of the three-dimensional point cloud reaches a preset scale value.

[0033] Please continue reading Figure 3In S5-S7, the first target iteration point is added to the three-dimensional point cloud to enrich the three-dimensional point cloud, and the normal vector and three-dimensional information of the first target iteration point are calculated, and the three-dimensional point cloud is re-triangulated until the points in the three-dimensional point cloud reach a preset scale value, and then the iteration is stopped.

[0034] In some embodiments, the calculated first target iteration point is placed in the three-dimensional point cloud, the pixels of the first target iteration point in each frame of the image can be determined according to the pinhole imaging model, and the coordinate value in the three-dimensional point cloud can be further determined according to the pixels.

[0035] In some embodiments, in order to avoid an infinite loop, a maximum number of iterations may be set. Once this number is reached, the iteration process will automatically stop, even if the point cloud scale or error assessment has not yet reached an ideal state. The maximum number of iterations is not limited in the embodiments of the present application.

[0036] Through this iterative triangulation method, the 3D point cloud model is gradually optimized to meet specific scale requirements while maintaining geometric accuracy and surface quality.

[0037] In the embodiment of the present application, since the preset light source and the camera are arranged at the same coordinate point, the position of the preset light source can be known by determining the posture of the camera during movement. Therefore, the technical problem of being unable to estimate the illumination direction of the reconstructed target surface in a moving state can be solved, thereby achieving the technical effect of improving the applicable scenarios of the photometric stereo method.

[0038] The camera and light source are arranged closely, and the problem of solving the light source direction is cleverly transformed into solving the camera position problem through their spatial position relationship. The camera's posture is obtained and then the constantly changing position of the light source during the movement is obtained. This effectively solves the problem of determining the light source position when using the photometric stereo method under moving conditions, and effectively improves the usage scenarios of the photometric stereo method.

[0039] In some embodiments, based on the calculation of the photometric stereo normal vector using the Lambertian model, the illumination intensity of a point on the reconstructed target surface has the following relationship:

[0040] in, Represents the illumination intensity at the target point being reconstructed. represents the albedo of the reconstructed target surface, represents the normal vector of the point, Indicates the lighting direction of the point. Since the normal vector is a unit vector, and They are integrated. In the embodiment of the present application, the camera and the light source are located at the same coordinate point, so the position of the light source is the position of the camera.

[0041] See also Figure 4 , Figure 4 A schematic diagram of a lighting model provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, point S is located on the surface of the reconstructed target, and its coordinates in the world coordinate system are , the positions of the kth camera and light source are expressed as , the light direction at point S for:

[0042] After the spatial coordinates of point S and the positions of the kth camera and light source are known, the illumination direction at point S can be calculated by combining the above calculation formula .

[0043] According to the near-field point light source distance attenuation model, the illumination intensity of the initial three-dimensional point in the first frame perspective is set to , the distance from the light source to the three-dimensional point is , then the distance from the camera to the i-th 3D point in the k-th frame is , then the light intensity at any specific point on the surface It can be expressed as:

[0044] After integrating the above formula, we can get the normal vector of each point The calculation formula is:

[0045] The near-field point light source distance attenuation model describes the law that the light intensity of a point light source decreases as the distance increases within a close range. The following are some common near-field point light source distance attenuation models: inverse law, attenuation exponential model, hyperbolic model and exponential law. In practical applications, the choice of model depends on the specific characteristics of the light source and the application scenario, and the embodiments of this application do not limit this.

[0046] See also Figure 5 , Figure 5 A schematic diagram of a method for constructing a scene provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, including: Step 201, construct at least one triangular facet according to the three-dimensional point cloud; wherein the vertices of the triangular facet are points in the three-dimensional point cloud.

[0047] See also Figure 6 , Figure 6 A schematic diagram of an iterative calculation process provided in an embodiment of the present application is shown in FIG. Figure 6As shown, in some embodiments, in the triangulation process, firstly, a large amount of three-dimensional point cloud data (i.e. Figure 6 The distribution of these points in space may be very irregular, so data structures such as octrees or K-dimensional trees can be used to quickly retrieve neighboring points and establish connections between points.

[0048] Select a suitable triangulation algorithm to construct triangular facets. In some embodiments, triangulation can be used to construct and generate a triangular mesh that is as uniform as possible, thereby avoiding too small or too large angles while maintaining the geometric shape. In triangulation, the circumscribed circle of each triangle will not contain other points in the point cloud, ensuring the quality and stability of the mesh. In some embodiments, triangular facets can also be constructed using methods such as von Neumann graphs, which are not specifically limited in the embodiments of the present application.

[0049] Step 202: determine the spatial coordinates of the centroid of each triangular facet according to the normal vectors of the vertices of each triangular facet.

[0050] For a triangle, its centroid is an important geometric feature point of the triangle. The basic principle of calculating the coordinates of the centroid is to add the coordinate values ​​of the three vertices of the triangle on each coordinate axis (X axis, Y axis, Z axis) respectively, and then divide the sum by 3 to get the spatial coordinates of the centroid.

[0051] Step 203, determining the normal vector of the center of gravity point according to the spatial coordinates of the center of gravity point.

[0052] In some embodiments, when calculating the normal vector of the centroid, the vector can be calculated by the average method, specifically, the normal vector of the centroid is approximated by taking the weighted average of the normal vectors of the three vertices of the triangle; it can also be based on a geometric calculation method, specifically, the normal vector of the triangle is taken, and the vector obtained by the vector cross product is perpendicular to the plane determined by the two vectors involved in the cross product, which just meets the property that the normal vector is perpendicular to the triangle. Specifically, in practical applications, the algorithm can be selected for calculation according to specific needs and determined triangle parameters, and the embodiments of the present application do not limit this.

[0053] Step 204, taking the normal vector of the centroid as the iteration direction, and determining the spatial coordinates of the iteration point according to the centroid and a preset step size.

[0054] In some embodiments, determining the spatial coordinates of the iteration point according to the center of gravity and the preset step size includes: The spatial coordinates of the iteration point are obtained by adding the product of the preset step size and the normal vector of the center of gravity to the spatial coordinates of the center of gravity.

[0055] See also Figure 7 , Figure 7 A triangulation schematic diagram provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the three vertices of the triangle are defined as , , , whose normal vectors correspond to According to the normal vectors of the three vertices, the coordinates of the center of gravity of the sphere G (i.e. Figure 7 middle ) = (A+B+C) / 3, that is, the coordinates of the center point are obtained. Represents the distance from the center of gravity to the three vertices, then the normal vector at the center of gravity is It can be expressed as:

[0056] in, is the distance from the centroid to vertex A, is the distance from the centroid to vertex B, is the distance from the centroid to the vertex C.

[0057] Center of gravity As the initial point, and the normal vector at the center of gravity is the search direction and sets the search step size , then the iteration point P (i.e. Figure 7 middle ) is expressed as:

[0058] In some embodiments, the search step size Can be set to one tenth of the shortest distance from the centroid to the three sides of the triangle; the normal vector of point P The calculation method and the normal vector at the center of gravity The calculation method is the same, and the present embodiment will not be repeated one by one. It should be noted that this description is only an exemplary description and is not a limitation on the specific step length, and the present embodiment does not limit this. Using this method, an accurate triangular patch model can be effectively constructed from the 3D point cloud, and by calculating the spatial coordinates and normal vector of the center of gravity point, the position and direction of the iteration point in space can be ensured to be accurately controllable, thereby improving the accuracy and surface quality of 3D model reconstruction.

[0059] See also Figure 8 , Figure 8 A schematic diagram of a flow chart of another method for constructing a scenario provided in an embodiment of the present application includes: Step 301 , calculating the photometric error corresponding to each iteration point according to the spatial coordinates of the vertices of each triangular facet and the spatial coordinates of the iteration point corresponding to each triangular facet.

[0060] The photometric error e(p) is used to determine the specific position of the iteration point P. Assume that the three-dimensional points A, B, C, and P are projected onto the pixel points in the pixel coordinate system respectively. , , , , the photometric error e(p) is expressed as:

[0061] When the pixel coordinates of the four points A, B, C, and P are known, the pixel coordinates of A, B, C, and P and the distances from the iteration point P to A, B, and C respectively are substituted into the calculation formula of the photometric error e(p) to calculate the photometric error of the iteration point P.

[0062] Step 302: among the photometric errors corresponding to each iteration point, determine the iteration point with the smallest photometric error value as the first target iteration point.

[0063] The formula calculates the sum of the squares of the photometric differences between the three reference points and then takes the square root to obtain the photometric error e(p). The smaller the error value, the better the photometric consistency of point P; the larger the error value, the worse the photometric consistency; therefore, among multiple iteration points, the iteration point with the smallest photometric error is determined as the first target iteration point.

[0064] By accurately calculating the photometric error corresponding to each iteration point, the matching degree between the iteration point and the reference point is evaluated, and the first target iteration point with the minimum photometric error value is determined, so that the 3D address of the first target iteration point is closer to the reconstructed target surface, the surface reconstruction quality of the 3D model is optimized, and the visual consistency under lighting conditions is ensured, thereby improving the accuracy and reliability of the final result.

[0065] In some embodiments, the step size in the iteration point P can be determined by finding the minimum value of e(p) The value of , finally determines the position of the iteration point P.

[0066] In some embodiments, before computing the normal vector of each point in the three-dimensional point cloud based on the camera pose and the three-dimensional point cloud of the reconstruction target, the method further comprises: Collecting raw data of the reconstructed target based on a preset light source and a camera; wherein the raw data includes appearance image data of the reconstructed target; In some embodiments, when collecting raw data of the reconstructed target, the camera is determined to be a forward projection device to ensure that each point on the image corresponds to the position of the reconstructed target surface. It is also determined that the light projected from the infinitely far light point to the reconstructed target is emitted, and the illumination angle and brightness of all points are kept consistent.

[0067] The camera pose in the world coordinate system and the 3D point cloud of the reconstructed target are calculated based on the original data of the reconstructed target, feature points are extracted from each image, and the feature descriptor of each feature point is calculated. These feature points can be points with good texture or edges in the image, such as corner points or edge points.

[0068] Feature Descriptor is a key technology used in the field of computer vision for image recognition and image matching. It aims to extract local feature information of key points in an image (such as corner points, edge points, etc.). This information can be used to describe the features of the image so as to match and identify feature points between different images.

[0069] In some embodiments, by comparing the feature descriptors between different images, the corresponding feature points are matched. A feature similarity measurement method, such as nearest neighbor matching or feature distance-based matching, can be used for matching. In practical applications, the method used for feature point matching can be determined according to actual needs, and the embodiments of the present application do not limit this.

[0070] Please continue reading Figure 3 ,like Figure 3 As described in S1-S2, in order to ensure the high quality of the collected image data, one or more light sources with known characteristics are set. These light sources can be continuous light sources, such as light emitting diodes (LED) lamps, or pulsed light sources, such as flash lamps; in some embodiments, the preset light source is a monochromatic light source. Specifically, the embodiments of the present application do not limit this.

[0071] Use cameras to take pictures of the reconstructed object from different angles and positions. These cameras can be SLR cameras, professional scanning equipment, or a series of synchronized cameras. During the acquisition process, the camera needs to capture multiple appearance image data of the target object, which will be used for subsequent 3D reconstruction.

[0072] The original data contains the color, texture and shape information of the reconstructed object. The quality of the image directly affects the accuracy of the reconstruction result, so it is necessary to minimize noise and distortion.

[0073] In some embodiments, before calculating the camera pose, the camera needs to be calibrated to obtain its internal parameters (such as focal length, principal point) and external parameters (such as position and orientation). By analyzing the collected image data, computer vision algorithms (such as feature matching, bundle adjustment) or simultaneous localization and mapping (SLAM) technology can be used to estimate the camera pose in the world coordinate system when each image is taken.

[0074] This method makes the reconstructed three-dimensional model highly consistent with the actual target in terms of geometric structure, providing a solid foundation for subsequent analysis and application.

[0075] In some embodiments, before placing the first target iteration point in the three-dimensional point cloud, the following steps are included: determining the pixel position of the first target iteration point in the original data based on the intrinsic parameter coefficient and the transformation matrix of the camera.

[0076] Assume that the coordinates of the iteration point P are , whose pixels in a certain frame image are , the camera internal parameter coefficient is A The transformation matrix of the camera's perspective relative to the image's perspective is: [RT], and the three-dimensional coordinates of point P can be obtained and pixel coordinates The corresponding relationship is:

[0077] The pixel position of point P in all image frames can be calculated.

[0078] In some embodiments, after calculating the normal vector of the first target iteration point based on the camera pose and the spatial coordinates of the first target iteration point, the method further includes: The normal vector of the first target iteration point is integrated to obtain the target space coordinates of the first target iteration point.

[0079] In some embodiments, this step further includes: calculating a normal vector of the first target iteration point based on the camera pose and the spatial coordinates of the first target iteration point.

[0080] The red, green and blue (RGB) color information in the image can be used to infer the lighting information of point P in each frame of pixels. The lighting direction at point P can be obtained based on the three-dimensional coordinates of point P and the position of the camera in each frame of the image. The normal vector of the point can be calculated based on the photometric stereo method, and the specific position of point P can be further determined by integrating the normal vector.

[0081] Placing the first target iteration point in the 3D point cloud includes: Place the first target iteration point at the target space coordinates in the 3D point cloud.

[0082] The problem of sparse reconstructed point cloud and insignificant reconstruction effect in the process of scene reconstruction is effectively solved through triangulation algorithm constraints and photometric stereo recalculation. It effectively expands the usage scenarios and enhances the expressiveness of this method in actual use. In addition, compared with other methods, the embodiment of the present application proposes that the computing resources required for the use of triangulation and photometric stereo are relatively small, and a relatively good model reconstruction effect can be achieved without huge resource consumption. It is more flexible and highly scalable, and has high practicality in the field of three-dimensional modeling.

[0083] In some embodiments, the method further includes: iteratively triangulating the three-dimensional point cloud with the first target iteration point added thereto, and placing the calculated second target iteration point in the three-dimensional point cloud.

[0084] In some embodiments, the camera collects raw data of the reconstructed target in any direction and at any angle.

[0085] The embodiments of the present application make it possible to apply the structure-from-motion method and the photometric stereo method to actual scenes with low resource consumption and fast and accurate reconstruction effects. In real life, there are many scenes that need to be reconstructed in real time, such as 360-degree panoramic images of cars. Such scenes require fast scene reconstruction. In the case of poor network, it is impossible to call for huge resources for scene reconstruction. At this time, the embodiments of the present application can become corresponding alternative applications.

[0086] In some embodiments, the accuracy of the reconstruction result is further improved by optimizing the camera pose and the position of the 3D points. In addition, sparseness can be performed, that is, deleting some repeated or poor quality feature points and camera poses.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0088] The present application also provides a scene construction device. Fig. 9 A schematic diagram of a structure of a scene construction device provided in an embodiment of the present application, such as Fig. 9 As shown, including: A first calculation unit 41 is used to calculate a normal vector of each point in the three-dimensional point cloud based on the camera posture and the three-dimensional point cloud of the reconstructed target; wherein the camera posture and the three-dimensional point cloud are calculated based on the preset light source at the same coordinate point and the data collected by the camera; A determination unit 42, configured to perform triangulation based on the three-dimensional point cloud and the normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point; A second calculation unit 43, configured to determine a first target iteration point according to a photometric error of at least one iteration point; The iterative calculation unit 44 is used to place the first target iteration point in the three-dimensional point cloud and iteratively triangulate the three-dimensional point cloud until the scale of the three-dimensional point cloud reaches a preset scale value.

[0089] In the embodiment of the present application, since the preset light source and the camera are arranged at the same coordinate point, the position of the preset light source can be known by determining the posture of the camera during movement. Therefore, the technical problem of being unable to estimate the illumination direction of the reconstructed target surface in a moving state can be solved, thereby achieving the technical effect of improving the applicable scenarios of the photometric stereo method.

[0090] Furthermore, in a possible implementation of the embodiment of the present application, the determining unit 42 is further configured to: Constructing at least one triangular facet according to the three-dimensional point cloud; wherein the vertices of the triangular facet are points in the three-dimensional point cloud; Determine the spatial coordinates of the centroid of each triangle according to the normal vector of the vertex of each triangle; Determine the normal vector of the center of gravity point according to the spatial coordinates of the center of gravity point; The normal vector of the center of gravity is used as the iteration direction, and the spatial coordinates of the iteration point are determined according to the center of gravity and the preset step size.

[0091] Furthermore, in a possible implementation of the embodiment of the present application, the determining unit 42 is further configured to: Calculate the photometric error corresponding to each iteration point according to the spatial coordinates of the vertices of each triangle and the spatial coordinates of the iteration point corresponding to each triangle; Among the photometric errors corresponding to each iteration point, the iteration point with the smallest photometric error value is determined as the first target iteration point.

[0092] Furthermore, in a possible implementation of the embodiment of the present application, the determining unit 42 is further configured to: The spatial coordinates of the iteration point are obtained by adding the product of the preset step size and the normal vector of the center of gravity to the spatial coordinates of the center of gravity.

[0093] Furthermore, in a possible implementation of the embodiment of the present application, as Fig.10 As shown, the device also includes: The acquisition unit 45 is used to acquire raw data of the reconstructed target based on a preset light source and a camera before the first calculation unit 41 calculates the normal vector of each point in the three-dimensional point cloud based on the camera pose and the three-dimensional point cloud of the reconstructed target; wherein the raw data includes appearance image data of the reconstructed target; The third calculation unit 46 is used to calculate the camera pose in the world coordinate system and the three-dimensional point cloud of the reconstructed target according to the original data of the reconstructed target.

[0094] Furthermore, in a possible implementation of the embodiment of the present application, the iterative calculation unit 44 is further configured to: The pixel position of the first target iteration point in the original data is determined based on the intrinsic parameter coefficients and the transformation matrix of the camera.

[0095] Furthermore, in a possible implementation of the embodiment of the present application, as Fig.10 As shown, the device also includes: A fourth calculation unit 47 is used for performing integral calculation on the normal vector of the first target iteration point after the iterative calculation unit 44 calculates the normal vector of the first target iteration point based on the camera pose and the spatial coordinates of the first target iteration point to obtain the target spatial coordinates of the first target iteration point; The iterative calculation unit 44 is further used for: Place the first target iteration point at the target space coordinates in the 3D point cloud.

[0096] Furthermore, in a possible implementation of the embodiment of the present application, as Fig.10 As shown, the device also includes: The fifth calculation unit 48 is used to calculate the normal vector of the first target iteration point based on the camera posture and the spatial coordinates of the first target iteration point after the iterative calculation unit 44 determines the pixel position of the first target iteration point in the original data based on the intrinsic parameter coefficient of the camera and the transformation matrix.

[0097] Furthermore, in a possible implementation of the embodiment of the present application, as Fig.10 As shown, the iterative calculation unit 44 is also used for: Iterative triangulation is performed on the three-dimensional point cloud to which the first target iteration point is added, and the calculated second target iteration point is placed in the three-dimensional point cloud.

[0098] Furthermore, in a possible implementation of the embodiment of the present application, the camera collects raw data of the reconstructed target in any direction and at any angle.

[0099] Furthermore, in a possible implementation of the embodiment of the present application, the preset light source is a monochromatic light source.

[0100] For the description of the features in the embodiment corresponding to the scene construction device, please refer to the relevant description of the embodiment corresponding to the scene construction method, which will not be repeated here.

[0101] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in the construction method embodiment of any of the above scenarios.

[0102] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of the construction method embodiment of any of the above scenarios when running.

[0103] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0104] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the construction method embodiment of any of the above scenarios are implemented.

[0105] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the construction method embodiment of any of the above scenarios are implemented.

[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0107] The above is a detailed introduction to a method and device for constructing a scenario, an electronic device, and a storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for constructing a scene, characterized in that: include: The normal vector of each point in the three-dimensional point cloud is calculated based on the camera pose and the three-dimensional point cloud of the reconstructed target; wherein the camera pose and the three-dimensional point cloud are calculated based on the preset light source at the same coordinate point and the data collected by the camera; Perform triangulation based on the three-dimensional point cloud and the normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point; Determine a first target iteration point according to the photometric error of the at least one iteration point; The first target iteration point is placed in the three-dimensional point cloud, and the three-dimensional point cloud is iteratively triangulated until the scale of the three-dimensional point cloud reaches a preset scale value.

2. The method for constructing a scene according to claim 1, characterized in that: The triangulation based on the three-dimensional point cloud and the normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point comprises: Constructing at least one triangular facet according to the three-dimensional point cloud; wherein the vertices of the triangular facet are points in the three-dimensional point cloud; Determine the spatial coordinates of the centroid of each triangular facet according to the normal vector of the vertex of each triangular facet; Determine the normal vector of the center of gravity point according to the spatial coordinates of the center of gravity point; The normal vector of the center of gravity is used as the iteration direction, and the spatial coordinates of the iteration point are determined according to the center of gravity and a preset step size.

3. The method for constructing a scene according to claim 2, characterized in that: Determining the first target iteration point according to the photometric error of the at least one iteration point comprises: Calculate the photometric error corresponding to each iteration point according to the spatial coordinates of the vertices of each triangular facet and the spatial coordinates of the iteration point corresponding to each triangular facet; Among the photometric errors corresponding to each of the iteration points, the iteration point with the smallest photometric error value is determined as the first target iteration point.

4. The method for constructing a scene according to claim 2, characterized in that: Determining the spatial coordinates of the iteration point according to the center of gravity and the preset step size includes: The spatial coordinates of the iteration point are obtained by adding the product of the preset step size and the normal vector of the center of gravity point to the spatial coordinates of the center of gravity point.

5. The method for constructing a scene according to claim 1, characterized in that: Before calculating the normal vector of each point in the three-dimensional point cloud based on the camera pose and the reconstruction target, the method further includes: Collecting raw data of the reconstructed target based on the preset light source and the camera; wherein the raw data includes appearance image data of the reconstructed target; The camera pose and the three-dimensional point cloud of the reconstructed target are calculated in a world coordinate system according to the original data of the reconstructed target.

6. The method for constructing a scene according to claim 5, characterized in that: Before placing the first target iteration point in the three-dimensional point cloud, the method further includes: The pixel position of the first target iteration point in the original data is determined based on the intrinsic parameter coefficients and the transformation matrix of the camera.

7. The method for constructing a scene according to claim 6, characterized in that: After determining the pixel position of the first target iteration point in the original data based on the intrinsic parameter coefficient and the transformation matrix of the camera, the method further includes: A normal vector of the first target iteration point is calculated based on the camera pose and the spatial coordinates of the first target iteration point.

8. The method for constructing a scene according to claim 7, characterized in that: After calculating the normal vector of the first target iteration point based on the camera pose and the spatial coordinates of the first target iteration point, the method further includes: Performing integral calculation on the normal vector of the first target iteration point to obtain the target space coordinates of the first target iteration point; Placing the first target iteration point in the three-dimensional point cloud comprises: The first target iteration point is placed at the target space coordinates in the three-dimensional point cloud.

9. The method for constructing a scene according to claim 8, characterized in that: The iterative triangulation of the three-dimensional point cloud until the scale of the three-dimensional point cloud reaches a preset scale value comprises: Iterative triangulation is performed on the three-dimensional point cloud to which the first target iteration point is added, and the calculated second target iteration point is placed in the three-dimensional point cloud.

10. The scene construction method according to claim 5, characterized in that: The camera collects raw data of the reconstructed target in any direction and at any angle.

11. The method for constructing a scene according to claim 1, characterized in that: The preset light source is a monochromatic light source.

12. A scene construction device, characterized in that: include: A first calculation unit is used to calculate a normal vector of each point in the three-dimensional point cloud based on a camera posture and a three-dimensional point cloud of a reconstructed target; wherein the camera posture and the three-dimensional point cloud are calculated based on a preset light source at the same coordinate point and data collected by the camera; A determination unit, configured to perform triangulation based on the three-dimensional point cloud and a normal vector of each point in the three-dimensional point cloud to obtain at least one iteration point; A second calculation unit, configured to determine a first target iteration point according to a photometric error of the at least one iteration point; The iterative calculation unit is used to place the first target iteration point in the three-dimensional point cloud and iteratively triangulate the three-dimensional point cloud until the scale of the three-dimensional point cloud reaches a preset scale value.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the scene construction method as claimed in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the scene construction method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the scene construction method according to any one of claims 1 to 11 are implemented.

Images