Viewpoint optimization method, device, equipment and medium for three-dimensional reconstruction

Abstract

The present application discloses a viewpoint optimization method, apparatus, equipment and medium for three-dimensional reconstruction. The method obtains point cloud data in a target area and converts the point cloud data into raster data; performs viewpoint sampling on each detection point in the raster data and clusters the sampled viewpoints to obtain a viewpoint cluster set corresponding to each detection point; constructs a cost function between the detection point and the viewpoint; according to the cost function, sequentially obtains the viewpoint with the best cost in each viewpoint cluster set, and adds the viewpoint with the best cost to the total viewpoint set. The effects of the present application are as follows: The present application gives priority to viewpoints that may produce high-quality point clouds by comprehensively considering the detection range of the sensor and combining the cost function, thereby being able to generate the optimal viewpoint in real time, prompting the robot to efficiently explore and reconstruct in a complex three-dimensional environment, thereby improving the overall task execution efficiency and data quality.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional environment reconstruction, and in particular to a viewpoint optimization method, apparatus, device and medium for three-dimensional reconstruction. Background Art

[0002] In autonomous systems such as mobile robots, unmanned vehicles, and drones, 3D spatial reconstruction technology is fundamental to achieving key tasks such as environmental perception, path planning, and object detection. Currently, mainstream 3D reconstruction methods typically rely on the acquisition and processing of point cloud data. These methods are widely used in areas such as SLAM (Simultaneous Localization and Mapping) and multi-view stereo reconstruction. To obtain complete and high-quality 3D reconstruction results, a large number of viewpoints are typically generated through random or regular sampling. These are then screened using preset evaluation metrics to determine the optimal viewpoint.

[0003] At present, the existing 3D reconstruction viewpoint evaluation methods have problems such as low processing efficiency and poor viewpoint generation when applied to dynamic environments. There is an urgent need for an improved solution that can generate efficient viewpoints in real time and fully consider the characteristics of various sensors. Summary of the Invention

[0004] The purpose of this application is to provide a viewpoint optimization method, device, equipment and medium for three-dimensional reconstruction, so as to solve the problems of low processing efficiency and poor viewpoint generation effect in the three-dimensional reconstruction viewpoint evaluation method when applied to dynamic environments in related technologies.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a viewpoint optimization method for three-dimensional reconstruction, comprising:

[0007] Acquire point cloud data within a target area and convert the point cloud data into raster data;

[0008] Performing viewpoint sampling on each detection point in the raster data, and clustering the sampled viewpoints to obtain a viewpoint cluster set corresponding to each detection point;

[0009] Construct the cost function between the detection point and the viewpoint;

[0010] According to the cost function, the viewpoint with the best cost in each viewpoint cluster set is obtained in turn, and the viewpoint with the best cost is added to the total viewpoint set.

[0011] Furthermore, the cost function is calculated using the following formula:

[0012]

[0013] Among them, r i is the viewpoint, p k is the detection point, f seen is the visibility gain function, f res is the image information gain function, f mov is the motion detection cost function.

[0014] Furthermore, the visibility gain function is calculated using the following formula:

[0015]

[0016] Among them, check(r i ,p k ) represents the detection point p k In r i There is no occlusion at the viewpoint, inrange(r i ,p k ) represents the detection point p k In r i Within viewpoint range.

[0017] Furthermore, the image information gain function is specifically constructed as follows:

[0018] Generate triangular facets based on the point cloud data, passing through the detection point p k Construct the normal vector of the triangle; construct a first vector, wherein the first vector is the detection point p k Point to viewpoint r i ; Obtain the image information gain function based on the normal vector and the first vector.

[0019] Furthermore, the image information gain function is calculated using the following formula:

[0020]

[0021] Wherein, α is the angle between the normal vector and the first vector, d get is the distance between the normal vector and the first vector, d detect is the effective range of the sensor.

[0022] Furthermore, the motion detection cost function is calculated using the following formula:

[0023] fmov(ri,pk)=‖(ri +1 -ri)+(ri-ri-1)‖ 2

[0024] Among them, r i+1 and r i-1 For ri adjacent viewpoints.

[0025] Furthermore, the step of sequentially obtaining the viewpoint with the best cost in each viewpoint cluster set according to the cost function includes:

[0026] According to the cost function, a cost evaluation process is performed on each viewpoint in each viewpoint cluster set, and a viewpoint with the best cost in each viewpoint cluster set is obtained in sequence.

[0027] In a second aspect, the present application further provides a viewpoint optimization device for three-dimensional reconstruction, comprising:

[0028] An acquisition module, configured to acquire point cloud data within a target area and convert the point cloud data into raster data;

[0029] A sampling processing module is used to perform clustering processing on the sampling points around each detection point in the raster data to obtain a viewpoint cluster set corresponding to each detection point;

[0030] Cost function construction module, used to construct the cost function between the detection point and the viewpoint;

[0031] The viewpoint optimization module is used to sequentially obtain the viewpoint with the best cost in each of the viewpoint cluster sets according to the cost function, and add the viewpoint with the best cost to the total viewpoint set.

[0032] In a third aspect, the present application also provides a computer electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the above-mentioned viewpoint optimization methods for three-dimensional reconstruction are implemented.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned viewpoint optimization methods for three-dimensional reconstruction are implemented.

[0034] The present application provides a viewpoint optimization method, apparatus, device, and medium for 3D reconstruction, which have the following effects:

[0035] This application generates the optimal viewpoint based on point cloud information, thereby maximizing the reconstruction accuracy. First, this method comprehensively considers the detection range of the sensor to ensure that the selected viewpoint can cover the widest possible unknown area; second, the algorithm introduces a dynamic cost function to fully consider the feasibility between viewpoints; finally, this method also combines the image information gain of the sensor to give priority to viewpoints that may produce high-quality point clouds. By combining these factors, this method can generate the optimal viewpoint in real time, enabling the robot to efficiently explore and reconstruct in a complex three-dimensional environment, thereby improving the overall task execution efficiency and data quality. This method is applicable to various types of autonomous systems, especially in application scenarios where high-precision three-dimensional modeling is required, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a viewpoint optimization method for three-dimensional reconstruction according to an embodiment of the present application;

[0037] Figure 2 is a schematic diagram of point cloud data in an embodiment of the present application;

[0038] Figure 3 is a schematic diagram of raster data in an embodiment of the present application;

[0039] Figure 4 is a schematic diagram of constructing a visibility gain function according to an embodiment of the present application;

[0040] Figure 5 Schematic diagram of constructing an image gain function according to an embodiment of the present application;

[0041] Figure 6 is a schematic diagram of constructing a dynamic cost function in an embodiment of the present application;

[0042] Figure 7 1 is a schematic structural diagram of a viewpoint optimization device for three-dimensional reconstruction according to an embodiment of the present application;

[0043] Figure 8 It is a structural diagram of a computer electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0046] In this application, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components or interactions between two components. Those skilled in the art will understand the specific meanings of these terms in this application based on specific circumstances.

[0047] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0048] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms "a", "the", and "the" used in one or more embodiments of the present application are also intended to include plural forms unless the context clearly indicates otherwise.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the template description herein are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when...".

[0051] Currently, existing related technologies often ignore the actual detection range of the sensor, the dynamic cost generated during motion, and the influence of image information gain when evaluating viewpoints, resulting in poor performance of the final generated viewpoints in practical applications.

[0052] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes in certain embodiments will not be repeated. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] Please refer to Figure 1 The present invention provides a viewpoint optimization method for three-dimensional reconstruction, which includes at least the following steps:

[0054] S10: Acquire point cloud data within the target area and convert the point cloud data into raster data.

[0055] Specifically, point cloud data can be obtained through various sensors (laser scanners and lidars, etc.). For example, a robot equipped with a lidar can obtain point cloud data along the way by continuously scanning the surrounding environment during movement.

[0056] It is understood that point cloud data is a set of vectors in a three-dimensional coordinate system. These vectors are usually represented in the form of points. Each point has three-dimensional space coordinates (X, Y, Z), and sometimes also contains other attribute information such as color, normal vector, reflection intensity, etc. Figure 2 .

[0057] It should be noted that after obtaining the point cloud data, it is necessary to convert the point cloud data into raster data to facilitate subsequent processing. Figure 3 .

[0058] Specifically, point cloud data can be converted into raster data by projection or interpolation.

[0059] S20 , performing viewpoint sampling on each detection point in the raster data, and performing clustering processing on the sampled viewpoints to obtain a viewpoint cluster set corresponding to each detection point.

[0060] Specifically, the viewpoints around each detection point in the raster data are sampled, and then the sampled viewpoints are clustered to obtain a viewpoint cluster set corresponding to each detection point. S30: Construct a cost function between the detection point and the viewpoint.

[0061] Please continue to refer to Figure 3 In one embodiment of the present application, the cost function between the detection point and the viewpoint is constructed as follows: Figure 3 The red one is the viewpoint. The set of points that need to be detected on the model to be reconstructed is P I ={p1,p2,...,p k}, assuming that the viewpoint set for 3D reconstruction of the object is R v ={r1,r2,...,r i}, the problem can be described as the viewpoint set R v For the reconstructed target detection point set P I To conduct high-speed and efficient exploration, in order to evaluate the viewpoint r i For detection point p k The detection effect of , defines the following cost function:

[0062]

[0063] Among them, r i is the viewpoint, p k is the detection point, f seen is the visibility gain function, f res is the image information gain function, f mov is the motion detection cost function.

[0064] It can be understood that for a single detection viewpoint, the optimization goal is as follows: the problem is modeled as continuously optimizing the viewpoint set R v And maximize each 3D reconstruction detection point p k Information gain:

[0065]

[0066] See also Figure 4 In a specific embodiment of the present application, the visibility gain function is specifically constructed as follows: Figure 4 , in which the red grid is the point to be detected p k , viewpoint 1, viewpoint 2, and viewpoint 3 are viewpoint sets R v The three points in f seenThe cost function is 1 if and only if the point to be detected is within the current viewpoint detection range d and is not blocked, and 0 in all other cases. According to the above conditions, the calculation formula of the visibility gain function is obtained:

[0067]

[0068]

[0069] Among them, check(r i ,p k ) represents the detection point p k In r i There is no occlusion at the viewpoint, inrange(r i ,p k ) represents the detection point p k In r i Within viewpoint range.

[0070] See also Figure 5 In a specific embodiment of the present application, the image information gain function is specifically constructed as follows:

[0071] Generate triangular facets based on the point cloud data, passing through the detection point p k Construct the normal vector of the triangle; construct a first vector, wherein the first vector is the detection point p k Point to viewpoint r i ; Obtain the image information gain function based on the normal vector and the first vector.

[0072] Specifically, the image information gain function is calculated using the following formula:

[0073]

[0074] Wherein, α is the angle between the normal vector and the first vector, d get is the distance between the normal vector and the first vector, d detect is the effective range of the sensor.

[0075] See also Figure 6 In a specific embodiment of the present application, the motion detection cost function is specifically constructed as follows. In order to prevent the infeasibility of viewpoints, the present embodiment fully considers the dynamic cost and requires that the motion transfer between adjacent viewpoints conforms to the robot kinematics. Based on the above constraints, the calculation formula of the motion detection cost function is obtained:

[0076] fmov(ri,pk)=‖(ri +1 -ri)+(ri-ri-1)‖ 2

[0077] Among them, r i+1 and r i-1 For r i adjacent viewpoints.

[0078] S40 , according to the cost function, sequentially obtaining the viewpoint with the best cost in each of the viewpoint cluster sets, and adding the viewpoint with the best cost to the total viewpoint set.

[0079] In a specific embodiment of the present invention, sequentially obtaining the viewpoint with the best cost in each of the viewpoint cluster sets according to the cost function includes:

[0080] According to the cost function, a cost evaluation process is performed on each viewpoint in each viewpoint cluster set, and a viewpoint with the best cost in each viewpoint cluster set is obtained in sequence.

[0081] Specifically, according to the cost function obtained in step S30, each viewpoint cluster set is subjected to cost evaluation processing in turn, thereby obtaining the optimal viewpoint in each viewpoint cluster set, and these optimal viewpoints are added to the total viewpoint set, which is the result of viewpoint optimization.

[0082] This application provides a viewpoint optimization method for 3D reconstruction, which has the following effects:

[0083] This application generates the optimal viewpoint based on point cloud information, thereby maximizing the reconstruction accuracy. First, this method comprehensively considers the detection range of the sensor to ensure that the selected viewpoint can cover the widest possible unknown area; second, the algorithm introduces a dynamic cost function to fully consider the feasibility between viewpoints; finally, this method also combines the image information gain of the sensor to give priority to viewpoints that may produce high-quality point clouds. By combining these factors, this method can generate the optimal viewpoint in real time, enabling the robot to efficiently explore and reconstruct in a complex three-dimensional environment, thereby improving the overall task execution efficiency and data quality. This method is applicable to various types of autonomous systems, especially in application scenarios where high-precision three-dimensional modeling is required, and has broad application prospects.

[0084] See also Figure 7 The present application further provides a viewpoint optimization device 200 for three-dimensional reconstruction, comprising:

[0085] An acquisition module 201 is used to acquire point cloud data within a target area and convert the point cloud data into raster data;

[0086] The sampling processing module 202 is used to perform clustering processing on the sampling points around each detection point in the raster data to obtain a viewpoint cluster set corresponding to each detection point;

[0087] The cost function construction module 203 is used to construct a cost function between the detection point and the viewpoint;

[0088] The viewpoint optimization module 204 is configured to sequentially obtain the viewpoint with the best cost in each of the viewpoint cluster sets according to the cost function, and add the viewpoint with the best cost to the total viewpoint set.

[0089] See also Figure 8 An embodiment of the present application also provides a computer electronic device 300, including a memory 303 and a processor 302, wherein the memory 303 stores a computer program, and when the processor executes the computer program, it implements the steps of any of the above-mentioned viewpoint optimization methods for three-dimensional reconstruction.

[0090] Specifically, the electronic device 300 includes: a transceiver 301, a bus interface and a processor 302, wherein the processor 302 is used to obtain point cloud data in the target area and convert the point cloud data into raster data; perform viewpoint sampling on each detection point in the raster data, and cluster the sampled viewpoints to obtain a viewpoint cluster set corresponding to each detection point; construct a cost function between the detection point and the viewpoint; according to the cost function, obtain the viewpoint with the best cost in each viewpoint cluster set in turn, and add the viewpoint with the best cost to the total viewpoint set.

[0091] In the embodiment of the present application, the electronic device 300 further includes: a memory 303. Figure 8 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits such as one or more processors represented by processor 302 and memory represented by memory 303. The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 301 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 may store data used by the processor 302 when performing operations.

[0092] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned viewpoint optimization methods for three-dimensional reconstruction are implemented.

[0093] In this embodiment, the computer-readable storage medium may be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0094] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not limiting, and thus other examples of the exemplary embodiments may have different values.

[0095] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0097] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0098] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a terminal device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0099] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A viewpoint optimization method for three-dimensional reconstruction, characterized in that: include: Acquire point cloud data within a target area and convert the point cloud data into raster data; Performing viewpoint sampling on each detection point in the raster data, and clustering the sampled viewpoints to obtain a viewpoint cluster set corresponding to each detection point; Construct a cost function between the detection point and the viewpoint, wherein the cost function is calculated using the following formula: Among them, r i is the viewpoint, p k is the detection point, f seen is the visibility gain function, f res is the image information gain function, α1 is the weight of the visibility gain function, f mov is the motion detection cost function, α2 is the weight of the motion detection cost function, wherein the motion detection cost function is calculated using the following formula: f mov (r i ,p k )=||(r i+1 -r i )+(r i -r i-1 )|| 2 Among them, r i+1 and r i-1 For r i adjacent viewpoints; According to the cost function, the viewpoint with the best cost in each viewpoint cluster set is obtained in turn, and the viewpoint with the best cost is added to the total viewpoint set.

2. The viewpoint optimization method according to claim 1, wherein: The visibility gain function is calculated using the following formula: Among them, check(r i ,p k ) represents the detection point p k In r i There is no occlusion at the viewpoint, inrange(r i ,p k ) represents the detection point p k In r i Within viewpoint range.

3. The viewpoint optimization method according to claim 1, wherein: The image information gain function is specifically constructed as follows: Generate triangular facets based on the point cloud data, passing through the detection point p k Constructing a normal vector of the triangle; Construct a first vector, where the first vector is the detection point p k Point to viewpoint r i ; The image information gain function is obtained according to the normal vector and the first vector.

4. The viewpoint optimization method according to claim 3, wherein: The image information gain function is calculated using the following formula: Wherein, α is the angle between the normal vector and the first vector, d get is the distance between the normal vector and the first vector, d detect is the effective range of the sensor.

5. The viewpoint optimization method according to claim 1, wherein: The step of sequentially obtaining the viewpoint with the best cost in each viewpoint cluster set according to the cost function includes: According to the cost function, a cost evaluation process is performed on each viewpoint in each viewpoint cluster set, and a viewpoint with the best cost in each viewpoint cluster set is obtained in sequence.

6. A viewpoint optimization device for three-dimensional reconstruction, characterized in that: include: An acquisition module, configured to acquire point cloud data within a target area and convert the point cloud data into raster data; A sampling processing module is used to perform clustering processing on the sampling points around each detection point in the raster data to obtain a viewpoint cluster set corresponding to each detection point; The cost function construction module is used to construct the cost function between the detection point and the viewpoint, wherein the cost function is calculated using the following formula: Among them, r i is the viewpoint, p k is the detection point, f seen is the visibility gain function, f res is the image information gain function, α1 is the weight of the visibility gain function, f mov is the motion detection cost function, α2 is the weight of the motion detection cost function, wherein the motion detection cost function is calculated using the following formula: f mov (r i ,p k )=||(r i+1 -r i )+(r i -r i-1 )|| 2 Among them, r i+1 and r i-1 For r i adjacent vision; The viewpoint optimization module is used to sequentially obtain the viewpoint with the best cost in each of the viewpoint cluster sets according to the cost function, and add the viewpoint with the best cost to the total viewpoint set.

7. A computer electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the viewpoint optimization method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the viewpoint optimization method according to any one of claims 1 to 5 are implemented.

Images