Pose optimization method and device and computer readable storage medium

Abstract

The embodiment of the invention discloses a pose optimization method and device and a computer readable storage medium, and the method comprises the steps: obtaining the point cloud data of a plurality of local spaces of a target scene through a track collection or multi-point collection mode, and carrying out the recognition of the point cloud data of the plurality of local spaces through the combination of common-view region matching and plane region extraction, the plurality of local spaces are jointly used as constraint conditions of pose optimization of the plurality of local spaces, pose optimization of the plurality of local spaces is realized, and a three-dimensional point cloud model of a target scene is obtained, so that pose optimization of the plurality of local spaces can be constrained under the conditions that local data is low in overlapping degree and the pose problem is difficult to solve, and the accuracy of pose optimization is improved. Therefore, the overall data of the target scene can be better aligned under the global coordinate system.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology and point cloud processing technology, and in particular to a posture optimization method, device and computer-readable storage medium. Background Art

[0002] Input data for spatial 3D reconstruction typically includes images captured by cameras and point clouds collected by lidar. Because the space being captured is often large in area and volume, the captured space is often broken down into multiple points or trajectories collected over a longer period of time. The data collected from each segmented area is often referred to as local data. Multiple local data require six degrees of freedom (3D translation and rotation) in space to be displayed and applied in a unified global coordinate system.

[0003] Traditional pose optimization algorithms primarily seek to match overlapping common view regions between different local data sets, and then calculate a more optimal pose based on these matching regions. However, these traditional pose optimization algorithms are not very suitable for pose optimization of local data sets with low overlap. If the overlap between local data sets is low, matching the common view regions is difficult, and pose optimization becomes difficult to solve. Summary of the Invention

[0004] In order to solve the technical problems in the related art, the embodiments of the present disclosure provide a posture optimization method, device and computer-readable storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a posture optimization method is provided, the method comprising:

[0006] Acquire multiple local point cloud images of a target scene, wherein the local point cloud images include point cloud data information of any local space of the target scene, and the local point cloud images are point cloud data of multiple local spaces obtained by performing trajectory acquisition or point acquisition of multiple points on the target scene by a first acquisition device, and the point cloud data of the multiple local spaces constitute a point cloud data structure of the global scene of the target scene;

[0007] Determining whether there is an overlapping common view area between adjacent local point cloud images;

[0008] If there are overlapping common view areas between adjacent local point cloud images, determining a matching relationship between the overlapping common view areas from the adjacent local point cloud images;

[0009] For the non-overlapping common view area between adjacent partial point cloud images, extracting each planar area belonging to the non-overlapping common view area in at least one partial point cloud image;

[0010] Based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas, optimizing the poses of the multiple local point cloud images;

[0011] Based on the multiple local point cloud images after the pose optimization, a three-dimensional point cloud model of the target scene is determined.

[0012] As an optional embodiment, for the non-overlapping common view area between adjacent partial point cloud images, extracting each planar area belonging to the non-overlapping common view area in at least one partial point cloud image includes:

[0013] For any of the local point cloud images, the local point cloud image is plane-divided according to the normal vectors and coordinate positions of the point clouds in the local point cloud image to obtain a plurality of first plane areas included in the non-overlapping common viewing area in the local point cloud image.

[0014] As an optional embodiment, for the non-overlapping common view area between adjacent partial point cloud images, extracting each planar area belonging to the non-overlapping common view area in at least one partial point cloud image further includes:

[0015] Acquire local image data of the target scene corresponding to the local point cloud image, the local image data being image data of the same subscene of the target scene as the local point cloud image, synchronously acquired by the second acquisition device when the first acquisition device acquires any local point cloud image of the target scene;

[0016] Performing element division on the local image data using a preset semantic segmentation algorithm to obtain a plurality of element images in the local image data;

[0017] Determining a mapping relationship between any one of the local point cloud images and a plurality of element images in the local image data corresponding thereto according to preset calibration parameters of the first acquisition device and the second acquisition device;

[0018] Based on the mapping relationship, a plurality of second planar areas of the non-overlapping common view area are extracted from any one of the local point cloud images.

[0019] As an optional embodiment, for any of the local point cloud images, plane division is performed on the local point cloud image according to the normal vectors and coordinate positions of the point clouds in the local point cloud image to obtain multiple first plane areas included in the non-overlapping common view area in the local point cloud image, including:

[0020] For any of the local point cloud images, determining the direction of the normal vector of the point cloud of the local point cloud image according to the normal vector distribution of the neighborhood points of the point cloud in the local point cloud image;

[0021] In the local point cloud image, determining whether the direction angle between the normal vectors of any two point clouds is less than a preset angle threshold, and whether the coordinate position distance between the two point clouds is less than a preset distance threshold;

[0022] When it is determined that the direction angle between the normal vectors of the two point clouds is less than a preset angle threshold, and the coordinate position distance is less than a preset distance threshold, the two point clouds are divided into the same plane area;

[0023] The point cloud of the non-overlapping common view area in the local point cloud image is traversed, and the above steps are repeated to obtain a plurality of first planar areas of the non-overlapping common view area in the local point cloud image.

[0024] As an optional embodiment, for any of the local point cloud images, plane division of the local point cloud image is performed according to the normal vectors and coordinate positions of the point clouds in the local point cloud image to obtain multiple first plane areas included in the non-overlapping common view area in the local point cloud image, further comprising:

[0025] In the case where it is determined that the direction angle of the normal vectors of the two point clouds is less than the preset angle threshold and the coordinate position distance is less than the preset distance threshold, after dividing the two point clouds into the same plane area, determining the size of the plane area and the total number of point clouds based on all point cloud data information of any plane area;

[0026] When the size of the plane area is smaller than a preset size threshold or the total number of point clouds in the plane area is smaller than a preset number threshold, the plane area is marked as an unstable area, and the first plane area does not include the unstable area.

[0027] As an optional embodiment, extracting a plurality of second planar areas of the non-overlapping common view area from any one of the local point cloud images based on the mapping relationship includes:

[0028] Based on the mapping relationship, determining point cloud data information of multiple planes corresponding to multiple element images in the local image data from any one of the local point cloud images;

[0029] determining, based on the classification of the plurality of element images, point cloud data information of a plane of interest from the point cloud data information of the plurality of planes;

[0030] Based on the point cloud data information of the plane of interest, a plurality of second plane areas of the non-overlapping common view area are extracted from the point cloud data information of the plurality of planes.

[0031] As an optional embodiment, extracting a plurality of second plane areas of the non-overlapping common view area from the point cloud data information of a plurality of planes based on the point cloud data information of the plane of interest includes:

[0032] Based on the point cloud data information of the plane of interest, filtering the point cloud data information of non-planes of interest from the local point cloud image to retain the point cloud data information of the plane of interest;

[0033] A plane fitting process is performed on the retained point cloud data information of the plane of interest using a preset random sampling consistency algorithm to obtain a plurality of second plane areas of the non-overlapping common view area.

[0034] As an optional embodiment, the performing pose optimization on the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas includes:

[0035] For any of the local point cloud images, determining the local point cloud images adjacent to the local point cloud image to obtain a local image pair;

[0036] Based on the extracted planar regions belonging to the non-overlapping common view area, determining, in each pair of partial images, an angle between normal vectors of the planar regions across the partial images;

[0037] Determining a geometric relationship state between each planar region across the partial image according to an angle between normal vectors of each planar region across the partial image;

[0038] The multiple local point cloud images are subjected to pose optimization according to the matching relationship between the mutually matching overlapping common view areas and the geometric relationship state between the respective planar areas across the local images.

[0039] As an optional embodiment, the performing pose optimization on the multiple local point cloud images based on the matching relationship between the mutually matching overlapping common view areas and the geometric relationship state between the respective planar areas across the local images includes:

[0040] The matching relationship between the overlapping common-view areas that match each other and the geometric relationship state between the various planar areas across the local images are input into a preset nonlinear optimization algorithm to solve the preset nonlinear optimization algorithm as constraints of the preset nonlinear optimization algorithm to obtain the pose optimization results of the multiple local point cloud images.

[0041] According to a second aspect of an embodiment of the present disclosure, a posture optimization device is provided, the device comprising:

[0042] a point cloud image acquisition module, configured to acquire a plurality of local point cloud images of a target scene, wherein the local point cloud images include point cloud data information of any local space of the target scene, and the local point cloud images are point cloud data of multiple local spaces obtained by performing trajectory acquisition or point acquisition of multiple points on the target scene by a first acquisition device, and the point cloud data of the multiple local spaces constitute a point cloud data structure of the global scene of the target scene;

[0043] a common view area determination module, configured to determine whether there is an overlapping common view area between adjacent local point cloud images;

[0044] a common view region extraction module, configured to determine a matching relationship between overlapping common view regions from adjacent local point cloud images if there are overlapping common view regions between adjacent local point cloud images;

[0045] a plane region extraction module, configured to extract, for a non-overlapping common view region between adjacent local point cloud images, each plane region belonging to the non-overlapping common view region in at least one local point cloud image;

[0046] A posture optimization module is used to optimize the postures of the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas;

[0047] A three-dimensional point cloud model determination module is used to determine the three-dimensional point cloud model of the target scene based on the multiple local point cloud images after the pose optimization.

[0048] As an optional embodiment, the plane region extraction module includes:

[0049] The first extraction unit is used to perform plane division on any of the local point cloud images according to the normal vector and coordinate position of the point cloud in the local point cloud image to obtain multiple first plane areas included in the non-overlapping common viewing area in the local point cloud image.

[0050] As an optional embodiment, the plane region extraction module further includes:

[0051] an image data acquisition unit, configured to acquire local image data of the target scene corresponding to the local point cloud image, wherein the local image data is image data of the same subscene of the target scene as the local point cloud image, which is synchronously acquired by the second acquisition device when the first acquisition device acquires any local point cloud image of the target scene;

[0052] an element image determining unit, configured to perform element division on the local image data using a preset semantic segmentation algorithm to obtain a plurality of element images in the local image data;

[0053] a mapping relationship determining unit, configured to determine a mapping relationship between any one of the local point cloud images and a plurality of element images in the local image data corresponding thereto, based on preset calibration parameters of the first acquisition device and the second acquisition device;

[0054] The second extraction unit is used to extract from any of the local point cloud images based on the mapping relationship.

[0055] A plurality of second planar areas of the non-overlapping common viewing area are obtained.

[0056] As an optional embodiment, the first extraction unit includes:

[0057] a first determining subunit, configured to determine, for any of the local point cloud images, a direction of a normal vector of a point cloud of the local point cloud image according to a normal vector distribution of neighborhood points of the point cloud of the local point cloud image;

[0058] a judgment subunit, configured to judge, in the local point cloud image, whether the direction angle between the normal vectors of any two point clouds is less than a preset angle threshold, and whether the coordinate position distance between the two point clouds is less than a preset distance threshold;

[0059] a plane division subunit, configured to divide the two point clouds into the same plane region when it is determined that the direction angle between the normal vectors of the two point clouds is less than a preset angle threshold and the coordinate position distance is less than a preset distance threshold;

[0060] The first plane extraction subunit is configured to traverse the point cloud of the non-overlapping common view area in the local point cloud image and repeat the above steps to obtain a plurality of first plane areas of the non-overlapping common view area in the local point cloud image.

[0061] As an optional embodiment, the first extraction unit further includes:

[0062] a second determining subunit, configured to, after dividing the two point clouds into the same plane area, determine the size of the plane area and the total number of point clouds based on all point cloud data information of any plane area, if it is determined that the direction angle between the normal vectors of the two point clouds is less than a preset angle threshold and the coordinate position distance is less than a preset distance threshold;

[0063] The marking subunit is used to mark the plane area as an unstable area when the size of the plane area is smaller than a preset size threshold or the total number of point clouds in the plane area is smaller than a preset number threshold, and the first plane area does not include the unstable area.

[0064] As an optional embodiment, the second extraction unit includes:

[0065] a third determining subunit, configured to determine, from any one of the local point cloud images based on the mapping relationship, point cloud data information of a plurality of planes corresponding to a plurality of element images in the local image data;

[0066] a fourth determining subunit, configured to determine, based on the classification of the plurality of element images, point cloud data information of a plane of interest from the point cloud data information of the plurality of planes;

[0067] The second plane extraction subunit is configured to extract a plurality of second plane areas of the non-overlapping common view area from the point cloud data information of the plurality of planes based on the point cloud data information of the plane of interest.

[0068] As an optional embodiment, the second plane extraction subunit is further configured to:

[0069] Based on the point cloud data information of the plane of interest, filtering the point cloud data information of non-planes of interest from the local point cloud image to retain the point cloud data information of the plane of interest;

[0070] A plane fitting process is performed on the retained point cloud data information of the plane of interest using a preset random sampling consistency algorithm to obtain a plurality of second plane areas of the non-overlapping common view area.

[0071] As an optional embodiment, the posture optimization module includes:

[0072] an image pair determining unit, configured to determine, for any of the local point cloud images, the local point cloud images adjacent thereto, to obtain a local image pair;

[0073] a plane angle determination unit, configured to determine, in each partial image pair, a normal vector angle of each plane region across the partial images based on the extracted plane regions belonging to the non-overlapping common view area;

[0074] a geometric relationship determining unit, configured to determine a geometric relationship state between each planar region across the partial image according to an angle between normal vectors of each planar region across the partial image;

[0075] A posture optimization unit is used to optimize the posture of the multiple local point cloud images according to the matching relationship between the overlapping common view areas that match each other and the geometric relationship state between the various planar areas across the local images.

[0076] As an optional embodiment, the posture optimization unit includes:

[0077] A solving subunit is used to input the matching relationship between the mutually matching overlapping common view areas and the geometric relationship state between the various planar areas across the local images into a preset nonlinear optimization algorithm, so as to solve the preset nonlinear optimization algorithm as the constraint conditions of the preset nonlinear optimization algorithm and obtain the pose optimization results of the multiple local point cloud images.

[0078] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0079] a memory for storing a computer program product;

[0080] The processor is configured to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0081] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect above is implemented.

[0082] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, implement the method described in the first aspect above.

[0083] The technical solution provided by the embodiments of the present disclosure obtains point cloud data of multiple local spaces of the target scene by trajectory acquisition or multi-point acquisition. For the point cloud data of the multiple local spaces, the combination of common view area matching and plane area extraction is used as the constraint conditions for the posture optimization of the multiple local spaces, so as to achieve posture optimization of the multiple local spaces and obtain a three-dimensional point cloud model of the target scene. This avoids the problem that some posture problems are ill-posed and difficult to solve due to the small overlapping common view areas between the point cloud data of the local spaces collected at different points or different trajectory segments. The geometric features of the plane area and other information are fully utilized, so that even when the local data have low overlap and the posture problem is difficult to solve, the posture optimization of the multiple local spaces can be constrained, so that the overall data of the target scene can be better aligned in the global coordinate system.

[0084] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0086] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0087] Figure 1 This is one of the flowcharts of a posture optimization method of an embodiment of the method disclosed herein.

[0088] Figure 2 This is the second flowchart of a posture optimization method according to an embodiment of the disclosed method.

[0089] Figure 3 This is the third flowchart of a posture optimization method according to an embodiment of the disclosed method.

[0090] Figure 4 This is one of the schematic diagrams of plane region extraction according to an embodiment of the method disclosed herein.

[0091] Figure 5 This is a fourth flowchart of a posture optimization method according to an embodiment of the method disclosed herein.

[0092] Figure 6 This is the second schematic diagram of plane area extraction according to an embodiment of the method disclosed herein.

[0093] Figure 7 This is the fifth flowchart of a posture optimization method of an embodiment of the method disclosed herein.

[0094] Figure 8 This is one of the structural block diagrams of a posture optimization device according to an embodiment of the present invention.

[0095] Figure 9 This is the second structural block diagram of a posture optimization device according to an embodiment of the device disclosed herein.

[0096] Figure 10 The present invention is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0097] In the reconstruction of a three-dimensional model of a house or a spatial scene, a collection device such as a lidar is often used to collect point cloud data information of the house or spatial scene, so as to realize the three-dimensional model reconstruction using the point cloud data information. However, during the process of collecting point cloud data information, the objects in the house or spatial scene are relatively complex and may occlude each other. Even if the overlapping common view areas between the local data collected by sampling different points or different trajectory segments are small, the matching between the local data is difficult to find or very rare, resulting in some pose problems being ill-posed and difficult to solve. For example, when collecting point cloud data information of a house with multiple rooms, if there is basically no common view area between the point cloud data information of different rooms and they are only connected by a small amount of data in the door area, although the poses of the point cloud data information of different rooms are correct, the global pose (i.e., the whole house) conflicts with each other, resulting in the pose of the house not being better aligned in the global coordinate system.

[0098] In response to the technical problems in the related art, the embodiments of the present disclosure provide a posture optimization method, device, and computer-readable storage medium to solve the related problems. The technical solutions of the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0099] Figure 1 This is one of the flow charts of a posture optimization method according to an embodiment of the disclosed method. Figure 1 As shown, a posture optimization method according to an embodiment of the present disclosure may include the following steps:

[0100] Step 101: Acquire multiple local point cloud images of a target scene.

[0101] Among them, the local point cloud image includes the point cloud data information of any local space of the target scene, which is the point cloud data of multiple local spaces obtained by performing trajectory collection or point collection of multiple points on the target scene by the first acquisition device. The point cloud data of multiple local spaces are spliced and fused to obtain the point cloud data structure of the global scene of the target scene.

[0102] In the embodiment of the present disclosure, the first acquisition device can be an image acquisition device such as a laser radar for collecting point cloud data. As an embodiment, the first acquisition device can be used to collect point cloud data information of the target scene at different points. For example, the first acquisition device is placed in a room of a house and fixed on a tripod. The first acquisition device is controlled to rotate 360° to collect point cloud data information. The rotation of 360° is, for example, the collection of one point, which is defined as a local point cloud image. As another embodiment, the first acquisition device can be used to collect point cloud data information of the target scene in a sampling trajectory acquisition manner. For example, the first acquisition device can be held in hand and the point cloud data information of the house can be collected in a mobile manner from any position of a house. In this way, the point cloud data information within a certain time period can be defined as a local point cloud image in chronological order.

[0103] Step 102 : Determine whether there is an overlapping common view area between adjacent local point cloud images.

[0104] In the disclosed embodiments, if the local point cloud images are collected using a point-by-point method, adjacent local point cloud images can be local point cloud images of two adjacent points. For example, if two rooms in a house are divided into two points, then the local point cloud images are of two adjacent rooms. If the local point cloud images are collected using a trajectory collection method, local point cloud images of two adjacent time periods can be determined as adjacent local point cloud images, for example, two local point cloud images of the time period t-t1 and the time period t1-t2.

[0105] As an optional embodiment of the present disclosure, a search tree algorithm can be used to determine whether adjacent local point cloud images have overlapping common view areas. For example, a KD tree (K-dimensional tree) is constructed for each local point cloud image, and then each point cloud of the first local point cloud image (assuming it is called local point cloud image A) is traversed, and the KD tree is used to find the nearest neighbor point in the point cloud of the second local point cloud image (assuming it is called local point cloud image B). By comparing the distance between the nearest neighbor points with the threshold, it is determined whether it belongs to the overlapping common view area, and the points that meet the conditions are saved, which is the common overlapping area of the two local point cloud images. For another example, first, a spatial octree is established for the local point cloud image cloud, and then all point clouds are traversed to query whether the corresponding point cloud is located in the grid of the octree. If it is located in the same grid, the point cloud is a point cloud of the overlapping common view area.

[0106] Step 103 : When overlapping common view areas exist, a matching relationship between the overlapping common view areas is determined from adjacent local point cloud images.

[0107] The matching relationship between overlapping common view areas can be determined by determining two planar areas in any two local point cloud images that belong to the overlapping common view area. For example, through the previous step, it is determined that the A1 planar area in the local point cloud image A and the B1 planar area in the local point cloud image B belong to the overlapping common view area. Then, the matching relationship of the two planar areas belonging to the overlapping common view area can be retained or marked, such as A1-B1. In this way, in the subsequent pose optimization, this matching relationship can be used to mutually constrain the poses of the two local point cloud images, making the poses of the two local point cloud images more accurate and the alignment effect better.

[0108] In some embodiments, the matching relationship between overlapping common view areas can also be a point cloud registration process for two local point cloud images. For example, the point cloud with the closest distance can be selected as the corresponding point in the two local point cloud images, and the rotation and translation transformation matrices can be solved through all corresponding point pairs. Then, the error between the two local point cloud images can be made smaller and smaller through continuous iteration to achieve the matching relationship between the overlapping common view areas (i.e., overlapping common view area matching).

[0109] Step 104 : For the non-overlapping common view area between adjacent partial point cloud images, extract each planar area belonging to the non-overlapping common view area in at least one partial point cloud image.

[0110] As mentioned above, some local point cloud images may not have overlapping common view areas, or there may be fewer overlapping common view areas. In this case, the embodiment of the present disclosure extracts planar areas from these areas (non-overlapping common view areas), thereby utilizing prior knowledge such as the geometric relationship between planar areas (for example, the vertical relationship, parallel relationship, etc. between planar areas).

[0111] In the embodiment of the present disclosure, the plane area may be extracted by using the geometric distribution of point cloud information or by using image semantic information.

[0112] Step 105 : performing pose optimization on the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas.

[0113] In the disclosed embodiment, the matching relationship between overlapping common view areas (i.e., overlapping common view area matching) and the prior knowledge of each planar area of the non-overlapping common view area can be used as constraints for the pose optimization of multiple local point cloud images to perform pose optimization and adjust the pose between multiple local point cloud images.

[0114] Step 106 : Determine a three-dimensional point cloud model of the target scene based on the multiple local point cloud images after pose optimization.

[0115] After the global pose optimization of multiple local point cloud images is completed, multiple local point cloud images with smaller mutual errors and more uniform pose matching can be obtained. The optimized multiple local point cloud images can be spliced and fused to obtain a three-dimensional point cloud model of the target scene, which can be used for applications such as the generation of floor plan vector diagrams, providing more accurate model data for subsequent applications.

[0116] The technical solution provided by the embodiments of the present disclosure obtains point cloud data of multiple local spaces of the target scene by trajectory acquisition or multi-point acquisition. For the point cloud data of the multiple local spaces, the combination of common view area matching and plane area extraction is used as the constraint conditions for the posture optimization of the multiple local spaces, so as to achieve posture optimization of the multiple local spaces and obtain a three-dimensional point cloud model of the target scene. This avoids the problem that some posture problems are ill-posed and difficult to solve due to the small overlapping common view areas between the point cloud data of the local spaces collected at different points or different trajectory segments. The geometric features of the plane area and other information are fully utilized, so that even when the local data have low overlap and the posture problem is difficult to solve, the posture optimization of the multiple local spaces can be constrained, so that the overall data of the target scene can be better aligned in the global coordinate system.

[0117] exist Figure 1 Based on the embodiment shown, Figure 2 Another method embodiment is provided. Figure 2 This is a second flow chart of a posture optimization method according to an embodiment of the disclosed method. Figure 2 As shown, Figure 1 The step 104 shown in FIG. 1 can be further implemented by the following steps:

[0118] Step 1041 : for any local point cloud image, plane division is performed on the local point cloud image according to the normal vector and coordinate position of the point cloud in the local point cloud image to obtain a plurality of first plane areas included in the non-overlapping common view area in the local point cloud image.

[0119] Furthermore, if Figure 3 As shown, step 1041 can be implemented as follows:

[0120] Step 1041a: for any local point cloud image, determine the direction of the normal vector of the point cloud according to the normal vector distribution of the neighborhood points of the point cloud.

[0121] In a local point cloud image, the distribution of neighborhood points of the point cloud can be analyzed to calculate the normal vector of the point cloud. For example, for any point cloud in the local point cloud image, multiple point clouds that are relatively close to it (for example, all point clouds within a range of 1 meter or half a meter) can be searched. These point clouds can form a shape area, and the average normal distribution of all point clouds within the shape area, or the overall normal distribution of the shape area, can be calculated. The average normal distribution or the overall normal distribution is determined as the direction of the normal vector of the point cloud.

[0122] Step 1041b: In the local point cloud image, determine whether the direction angle between the normal vectors of any two point clouds is less than a preset angle threshold, and whether the coordinate position distance between the two point clouds is less than a preset distance threshold.

[0123] After determining the directions of the normal vectors of all point clouds in the local point cloud image, it is possible to further divide some point clouds with similar normal vector directions and adjacent positions into the same plane area based on the directions of the normal vectors. In this step, the angle threshold and the distance threshold can be pre-set, wherein the angle threshold is, for example, zero, that is, the normal vectors of the two point clouds are in the same direction, and the distance threshold is, for example, between 1cm-5cm. It is understandable that the angle threshold and the distance threshold provided here are only examples and do not limit the embodiments of the present disclosure. In actual applications, they can be set according to actual conditions, such as the angle threshold is zero or within the range of ±0.5°, etc., and it can be assumed that the normal vectors of the two point clouds are in the same direction, and the distance threshold can also be, for example, less than 1cm or less than 5cm.

[0124] Step 1041c: When it is determined that the direction angle of the normal vectors of the two point clouds is less than a preset angle threshold and the coordinate position distance is less than a preset distance threshold, the two point clouds are divided into the same plane area.

[0125] In the previous step, if it is determined that the direction angle of the normal vectors of the two point clouds is less than the preset angle threshold and the coordinate position distance is less than the preset distance threshold, the two point clouds can be divided into the same plane area.

[0126] Step 1041d: traverse the point clouds of the non-overlapping common view areas in the local point cloud image, and repeat the above steps to obtain multiple first planar areas of the non-overlapping common view areas in the local point cloud image.

[0127] By traversing all point clouds in the local point cloud image and repeatedly executing steps 1041a to 1041c, the point clouds in the local point cloud image can be divided into plane areas to obtain the first plane area contained in the local point cloud image.

[0128] by Figure 4 For example, Figure 4 Figure A is a local point cloud image, and Figure B is a multiple plane areas after the plane area is divided and extracted, such as plane areas 1, 2, and 3 in the figure.

[0129] In some other embodiments, step 1041 may further include the following steps:

[0130] After determining that the angle between the normal vectors of the two point clouds is less than a preset angle threshold and the distance between their coordinate positions is less than a preset distance threshold, the two point clouds are divided into the same planar region. The size and total number of points in any planar region can then be determined based on all the point cloud data information in that planar region. If the size of the planar region is less than a preset size threshold or the total number of points in the planar region is less than a preset number threshold, the planar region is marked as an unstable region, and the first planar region does not include the unstable region. This ensures that all first planar regions are stable, eliminating noise and improving data quality. Specifically, the size of the planar region and the total number of points in the planar region are used to exclude unstable regions that are too small. For example, a planar region is a two-dimensional shape. Based on prior knowledge, its size should generally be greater than 1m*1m. If it is smaller than this size, it is considered an unstable region. Alternatively, based on the acquisition frequency of the first acquisition device, the total number of point clouds in a planar region is generally at least 1,000. If it is smaller than this number, it is considered an unstable region.

[0131] Step 1042: Acquire local image data of the target scene corresponding to the local point cloud image.

[0132] The local image data is the image data of the same sub-scene of the target scene that is synchronously acquired by the second acquisition device when the first acquisition device acquires any local point cloud image of the target scene. Specifically, when acquiring the same spatial image, the first acquisition device and the second acquisition device are used to acquire images at the same time, wherein the first acquisition device (such as a lidar) is used to acquire point cloud data information of the space, and the second acquisition device (such as a camera) is used to acquire, for example, two-dimensional image data of the space. In this way, a local point cloud image and local image data of the same space (under the target scene) are obtained. The local image data acquired by the second acquisition device can be used as a data supplement to the local point cloud image acquired by the first acquisition device, so that the robustness of the spatial image (or the image of the target scene) is better and a better image data foundation is provided.

[0133] Step 1043 , dividing the local image data into elements using a preset semantic segmentation algorithm to obtain multiple element images in the local image data.

[0134] In this step, any local image data can be used as the input of the preset semantic segmentation algorithm, so as to output a segmentation map containing various pixel category labels through the processing of the semantic segmentation algorithm. A pixel category label can correspond to an element image, and the element image can be understood as the classification of the objects or planes contained in the local image data. In the embodiment of the present disclosure, for example, after the local image data is processed by the preset semantic segmentation algorithm, different classifications of element images such as "ceiling", "ground", and "wall" are obtained. It can be understood that the preset semantic segmentation algorithm includes, but is not limited to, FCN (Fully Convolutional Networks), U-Net, DeepLab, SegNet, etc., and the embodiment of the present disclosure is not limited to this.

[0135] Step 1044 : Determine a mapping relationship between any local point cloud image and a plurality of element images in the local image data corresponding thereto according to preset calibration parameters of the first acquisition device and the second acquisition device.

[0136] The preset calibration parameters of the first acquisition device and the second acquisition device include, for example, but are not limited to external parameters such as rotation matrix (for example, used to describe the rotation of the lidar coordinate system relative to the camera or camera coordinate system) and translation vector (for example, used to describe the translation of the lidar coordinate system relative to the camera or camera coordinate system). Based on these parameters, the mapping of the three-dimensional point cloud in the local point cloud image collected by the first acquisition device and the two-dimensional image in the local image data can be achieved, that is, the three-dimensional point cloud in the local point cloud image collected by the first acquisition device is mapped to the semantic information on the two-dimensional image, and the mapping between the three-dimensional point cloud in the local point cloud image and multiple element images in the local image data is achieved, so as to determine to which element image in the local image data the three-dimensional point cloud in the local point cloud image belongs.

[0137] Step 1045 : extracting a plurality of second planar regions of non-overlapping common view areas from any one of the local point cloud images based on the mapping relationship.

[0138] As an optional embodiment, Figure 5 As shown, step 1045 can be further implemented as follows:

[0139] Step 1045a: Based on the mapping relationship, determine point cloud data information of multiple planes corresponding to multiple element images in the local image data from any local point cloud image.

[0140] Step 1045b: Determine the point cloud data information of the plane of interest from the point cloud data information of the multiple planes based on the classification of the multiple element images.

[0141] Step 1045c: extracting a plurality of second plane regions of non-overlapping common view areas from the point cloud data information of the plurality of planes based on the point cloud data information of the plane of interest.

[0142] After determining the mapping relationship in the previous step 1044, in this embodiment, the plane area to which the point cloud in the local point cloud image belongs can be further determined based on the mapping relationship, for example, determining that a certain point cloud corresponds to a plane such as a ceiling, a ground or a door frame in the local image data. In addition, the element image obtained based on the preset semantic segmentation processing on the local image data can be further classified, for example, into areas of interest and areas of non-interest. In this way, the plane area to which the point cloud in the local point cloud image belongs can be further distinguished, such as the point cloud of the area of interest and the point cloud of the area of non-interest. Finally, multiple second plane areas are extracted based on the classified point cloud.

[0143] Furthermore, specifically, based on the point cloud data information of the plane of interest, the point cloud data information of the non-interest planes can be filtered from the local point cloud image, thereby retaining the point cloud data information of the plane of interest. Then, a plane fitting process is performed on the retained point cloud data information of the plane of interest using a preset random sampling consensus algorithm to obtain multiple second plane regions of non-overlapping common view areas. The preset random sampling consensus algorithm includes, but is not limited to, the RANSAC algorithm.

[0144] In this embodiment, step 1041 is a plane extraction using the geometric distribution of point cloud information, while steps 1042 to 1045 are point cloud plane extraction using image semantic information. In the disclosed embodiment, plane extraction can be achieved using only step 1041 or only steps 1042 to 1045, or the two parts can be combined to achieve plane extraction. As a preferred solution, in order to make the extracted plane area more complete, the plane extraction method using the geometric distribution of point cloud information and the point cloud plane extraction method using image semantic information are combined, that is, step 1041 and steps 1042 to 1045 are combined to extract a more complete plane area of the local point cloud image, provide a more complete data basis for posture optimization, and improve the model robustness of posture optimization.

[0145] Figure 6 This is a second schematic diagram of plane region extraction according to an embodiment of the method disclosed herein. Figure 6 Shown Figure 5 The figure shows a schematic diagram of point cloud plane extraction using image semantic information. Figure 6A is a local point cloud image, B is the semantics of each block (element image) obtained after the local image data is processed by a preset semantic segmentation algorithm, and C is multiple planar areas extracted based on the semantics in Figure B (such as the areas marked 1, 2, 3, 4, 5, 6, 7, and 8 in the figure), such as the areas corresponding to 1 and 2 in the figure.

[0146] Based on the above embodiments, the present disclosure also provides the following Figure 7 In the embodiment shown, 7 is a flowchart of a posture optimization method according to an embodiment of the present disclosure. Figure 7 As shown, step 105 can be implemented by the following steps:

[0147] Step 1051: For any local point cloud image, determine the local point cloud images adjacent to it to obtain a local image pair.

[0148] The local point cloud image in the above steps and the plane areas extracted from each local point cloud image are used as input, wherein the plane areas extracted between each local point cloud image may or may not overlap, and then a global spatial search tree (such as an octree, a KD tree, etc.) is used to search for the neighboring local point cloud images of any local point cloud image. For example, the local point cloud image has an initial pose, that is, the pose data of the acquisition device. The acquisition device will record it according to the acquisition order or point position. For example, any target scene has several points, and there are as many pose data as there are. The global spatial search tree can search for the neighboring local point cloud images of each local point cloud image only in the same or close quadrants based on the initial pose data. Finally, every two adjacent local point cloud images constitute a local image pair, such as local image pair A and B, local image pair A and C, etc.

[0149] Step 1052 : Based on the extracted planar regions belonging to the non-overlapping common view area, in any pair of partial images, determine the normal vector angles of the planar regions across the partial images.

[0150] For any pair of partial images, calculate the normal vector angles between the plane areas across the partial images. For example, for a pair of partial images A and B, A includes plane areas A1, A2, A3 and A4, and B includes plane areas B1, B2, B3 and B4. Then calculate the normal vector angles of all plane areas across the images, such as A1 and B1, A1 and B2, A1 and B3, A1 and B4...A4 and B1, A4 and B2, A4 and B3, A4 and B4. For example, it is known that the normal vectors of any two plane areas are respectively and The angle between the normal vectors of two plane regions can be calculated using the following formula:

[0151]

[0152] Step 1053 : determining the geometric relationship between the various planar regions across the partial image according to the normal vector angles of the various planar regions across the partial image.

[0153] For example, as a priori knowledge of plane areas, there is a certain geometric relationship between plane areas, so the geometric relationship between two plane areas can be determined by the angle between their normal vectors. For example, if the angle between the normal vectors of two plane areas is approximately 0 degrees, then the geometric relationship state of the two plane areas is "parallel" (meaning that the two surfaces are oriented in the same direction); if the angle between the normal vectors of two plane areas is approximately 180 degrees, then the geometric relationship state of the two plane areas is "antiparallel" (meaning that the two surfaces are oriented in the opposite direction); if the angle between the normal vectors of two plane areas is approximately 90 degrees, then the geometric relationship state of the two plane areas is "orthogonal" (meaning that the structural planes of the building are orthogonal); and if the angle between the normal vector of any plane area and the direction of gravity is approximately 90 degrees, then the plane area is determined to be "gravity perpendicular" (meaning that the normal vector of the plane area is perpendicular to the direction of gravity).

[0154] Step 1054 : performing pose optimization on the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the geometric relationship between the various planar areas across the local images.

[0155] Among them, step 1054 is specifically implemented by inputting the matching relationship between the mutually matching overlapping common-view areas and the geometric relationship state between the various planar areas across the local images into the preset nonlinear optimization algorithm, so as to solve the preset nonlinear optimization algorithm as the constraint conditions of the preset nonlinear optimization algorithm and obtain the pose optimization results of multiple local point cloud images.

[0156] In the embodiment of the present disclosure, the matching relationship between overlapping common view areas and the geometric relationship state between various planar areas across local images can be used as constraints to perform pose optimization solution. Exemplarily, for example, bundle adjustment is used as a nonlinear optimization problem algorithm to perform pose optimization, and the matching relationship between overlapping common view areas and the geometric relationship state between various planar areas across local images are input as feature point matching results of the bundle adjustment, so that the pose of the first acquisition device is estimated by the local point cloud image sequence and the feature point matching results. Then, for the point cloud of the local point cloud image, a light beam is formed by emitting light from the optical center of the first acquisition device corresponding to each view of the local point cloud image and passing through the pixel position of the point cloud in the image. The reprojection error is minimized by adjusting the pose of the first acquisition device, and then a least squares problem is constructed, all projection error terms are summed, and the optimal pose of the first acquisition device is solved by the least squares method. Further, an optimization algorithm (such as Ceres Solver) is used to perform bundle adjustment, and the coordinates of the pose of the first acquisition device and the three-dimensional point cloud of the local point cloud image are adjusted until the convergence condition is reached to complete the pose optimization of multiple local point cloud images. It can be understood that bundle adjustment is only a nonlinear optimization algorithm for achieving posture optimization. The embodiments of the present disclosure are only used as an example and do not constitute a limitation to the present disclosure. In actual applications, other nonlinear optimization algorithms can also be used to achieve posture optimization.

[0157] In summary, the technical solution provided by the embodiments of the present disclosure obtains point cloud data of multiple local spaces of the target scene through trajectory acquisition or multi-point acquisition. For the point cloud data of the multiple local spaces, the combination of common view area matching and plane area extraction is used as the constraint conditions for the posture optimization of the multiple local spaces, so as to achieve posture optimization of the multiple local spaces and obtain a three-dimensional point cloud model of the target scene. This avoids the problem that some posture problems are ill-posed and difficult to solve due to the small overlapping common view areas between the point cloud data of the local spaces collected at different points or different trajectory segments. The geometric features of the plane area and other information are fully utilized, so that even when the local data have low overlap and the posture problem is difficult to solve, the posture optimization of multiple local spaces can be constrained, so that the overall data of the target scene can be better aligned in the global coordinate system.

[0158] Correspondingly, the embodiments of the present disclosure also provide device embodiments corresponding to the aforementioned method embodiments. The device embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0159] Figure 8 This is one of the structural block diagrams of a posture optimization device according to an embodiment of the present invention. Figure 8As shown, a posture optimization device provided by an embodiment of the present disclosure may include a point cloud image acquisition module 801, a common view area determination module 802, a common view area extraction module 803, a plane area extraction module 804, a posture optimization module 805 and a three-dimensional point cloud model determination module 806, wherein:

[0160] The point cloud image acquisition module 801 is used to acquire multiple local point cloud images of the target scene, wherein the local point cloud images include point cloud data information of any local space of the target scene. The local point cloud images are point cloud data of multiple local spaces obtained by performing trajectory acquisition or point acquisition of multiple points of the target scene by a first acquisition device. The point cloud data of the multiple local spaces constitute the point cloud data structure of the global scene of the target scene.

[0161] A common view area determination module 802 is configured to determine whether there is an overlapping common view area between adjacent local point cloud images;

[0162] a common view region extraction module 803 for determining a matching relationship between overlapping common view regions from adjacent local point cloud images if there are overlapping common view regions between adjacent local point cloud images;

[0163] The plane region extraction module 804 is configured to extract, from the non-overlapping common view region between adjacent local point cloud images, each plane region belonging to the non-overlapping common view region in at least one local point cloud image;

[0164] A pose optimization module 805 is configured to optimize the poses of the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas;

[0165] The three-dimensional point cloud model determination module 806 is used to determine the three-dimensional point cloud model of the target scene based on the multiple local point cloud images after the pose optimization.

[0166] The technical solution provided by the embodiments of the present disclosure obtains point cloud data of multiple local spaces of the target scene by trajectory acquisition or multi-point acquisition. For the point cloud data of the multiple local spaces, the combination of common view area matching and plane area extraction is used as the constraint conditions for the posture optimization of the multiple local spaces, so as to achieve posture optimization of the multiple local spaces and obtain a three-dimensional point cloud model of the target scene. This avoids the problem that some posture problems are ill-posed and difficult to solve due to the small overlapping common view areas between the point cloud data of the local spaces collected at different points or different trajectory segments. The geometric features of the plane area and other information are fully utilized, so that even when the local data have low overlap and the posture problem is difficult to solve, the posture optimization of the multiple local spaces can be constrained, so that the overall data of the target scene can be better aligned in the global coordinate system.

[0167] exist Figure 8 Based on the embodiments shown, the present disclosure also provides Figure 9 The embodiment shown. Figure 9 This is a second structural block diagram of a posture optimization device according to an embodiment of the present invention. Figure 9 As shown, in Figure 8 On the basis of the embodiment shown, the posture optimization device provided in the embodiment of the present disclosure may further include:

[0168] The plane region extraction module 804 may further include:

[0169] The first extraction unit 8041 is used to perform plane division on any of the local point cloud images according to the normal vector and coordinate position of the point cloud in the local point cloud image, and obtain multiple first plane areas included in the non-overlapping common view area in the local point cloud image.

[0170] An image data acquisition unit 8042 is configured to acquire local image data of the target scene corresponding to the local point cloud image, wherein the local image data is image data of the same subscene of the target scene as the local point cloud image, which is acquired synchronously by the second acquisition device when the first acquisition device acquires any local point cloud image of the target scene;

[0171] The element image determining unit 8043 is configured to perform element segmentation on the local image data using a preset semantic segmentation algorithm to obtain a plurality of element images in the local image data;

[0172] a mapping relationship determining unit 8044, configured to determine a mapping relationship between any one of the local point cloud images and a plurality of element images in the local image data corresponding thereto, based on preset calibration parameters of the first acquisition device and the second acquisition device;

[0173] The second extraction unit 8045 is configured to extract a plurality of second planar regions of the non-overlapping common view area from any one of the local point cloud images based on the mapping relationship.

[0174] The pose optimization module 805 may further include:

[0175] An image pair determining unit 8051 is configured to determine, for any of the local point cloud images, the local point cloud images adjacent to the local point cloud image to obtain a local image pair;

[0176] a plane angle determination unit 8052 for determining, in each partial image pair, a normal vector angle of each plane region across the partial images based on the extracted plane regions belonging to the non-overlapping common view area;

[0177] a geometric relationship determining unit 8053, configured to determine a geometric relationship state between each planar region across the partial image according to an angle between normal vectors of each planar region across the partial image;

[0178] The posture optimization unit 8054 is used to optimize the posture of the multiple local point cloud images according to the matching relationship between the overlapping common view areas that match each other and the geometric relationship state between the various planar areas across the local images.

[0179] As an optional embodiment of the present disclosure, the first extraction unit 8041 may include:

[0180] a first determining subunit, configured to determine, for any of the local point cloud images, a direction of a normal vector of a point cloud of the local point cloud image according to a normal vector distribution of neighborhood points of the point cloud in the local point cloud image;

[0181] a judgment subunit, configured to judge, in the local point cloud image, whether the direction angle between the normal vectors of any two point clouds is less than a preset angle threshold, and whether the coordinate position distance between the two point clouds is less than a preset distance threshold;

[0182] a plane division subunit, configured to divide the two point clouds into the same plane region when it is determined that the direction angle between the normal vectors of the two point clouds is less than a preset angle threshold and the coordinate position distance is less than a preset distance threshold;

[0183] The first plane extraction subunit is configured to traverse the point cloud of the non-overlapping common view area in the local point cloud image and repeat the above steps to obtain a plurality of first plane areas of the non-overlapping common view area in the local point cloud image.

[0184] As an optional embodiment of the present disclosure, the first extraction unit 8041 may further include:

[0185] a second determining subunit, configured to, after dividing the two point clouds into the same plane area, determine the size of the plane area and the total number of point clouds based on all point cloud data information of any plane area, if it is determined that the direction angle between the normal vectors of the two point clouds is less than a preset angle threshold and the coordinate position distance is less than a preset distance threshold;

[0186] The marking subunit is used to mark the plane area as an unstable area when the size of the plane area is smaller than a preset size threshold or the total number of point clouds in the plane area is smaller than a preset number threshold, and the first plane area does not include the unstable area.

[0187] As an optional embodiment of the present disclosure, the second extraction unit 8045 may include:

[0188] a third determining subunit, configured to determine, from any one of the local point cloud images based on the mapping relationship, point cloud data information of a plurality of planes corresponding to a plurality of element images in the local image data;

[0189] a fourth determining subunit, configured to determine, based on the classification of the plurality of element images, point cloud data information of a plane of interest from the point cloud data information of the plurality of planes;

[0190] The second plane extraction subunit is configured to extract a plurality of second plane areas of the non-overlapping common view area from the point cloud data information of the plurality of planes based on the point cloud data information of the plane of interest.

[0191] Furthermore, the second plane extraction subunit is further configured to:

[0192] Based on the point cloud data information of the plane of interest, filtering the point cloud data information of non-planes of interest from the local point cloud image to retain the point cloud data information of the plane of interest;

[0193] A plane fitting process is performed on the retained point cloud data information of the plane of interest using a preset random sampling consistency algorithm to obtain a plurality of second plane areas of the non-overlapping common view area.

[0194] As an optional embodiment of the present disclosure, the posture optimization unit 8054 may include:

[0195] A solving subunit is used to input the matching relationship between the mutually matching overlapping common view areas and the geometric relationship state between the various planar areas across the local images into a preset nonlinear optimization algorithm, so as to solve the preset nonlinear optimization algorithm as the constraint conditions of the preset nonlinear optimization algorithm and obtain the pose optimization results of the multiple local point cloud images.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0197] Below, reference Figure 10 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be either or both of the first device and the second device, or a standalone device independent of them, and the standalone device may communicate with the first device and the second device to receive collected input signals from them.

[0198] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0199] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0200] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0201] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the posture optimization method of each embodiment of the present disclosure described above and / or other desired functions.

[0202] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0203] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0204] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0205] Of course, to simplify, Figure 10 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0206] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the posture optimization method according to various embodiments of the present disclosure described in the above part of this specification.

[0207] The computer program product may be written in any combination of one or more programming languages to implement the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0208] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the posture optimization method according to various embodiments of the present disclosure described in the above part of this specification.

[0209] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0210] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0211] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0212] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0213] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0214] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0215] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0216] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A posture optimization method, characterized in that: The method comprises: Acquire multiple local point cloud images of a target scene, wherein the local point cloud images include point cloud data information of any local space of the target scene, and the local point cloud images are point cloud data of multiple local spaces obtained by performing trajectory acquisition or point acquisition of multiple points on the target scene by a first acquisition device, and the point cloud data of the multiple local spaces constitute a point cloud data structure of the global scene of the target scene; Determining whether there is an overlapping common view area between adjacent local point cloud images; If there are overlapping common view areas between adjacent local point cloud images, determining a matching relationship between the overlapping common view areas from the adjacent local point cloud images; For the non-overlapping common view area between adjacent partial point cloud images, extracting each planar area belonging to the non-overlapping common view area in at least one partial point cloud image; Based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas, optimizing the poses of the multiple local point cloud images; Based on the multiple local point cloud images after the pose optimization, a three-dimensional point cloud model of the target scene is determined.

2. The method according to claim 1, characterized in that For the non-overlapping common view area between adjacent partial point cloud images, extracting each planar area belonging to the non-overlapping common view area in at least one partial point cloud image includes: For any of the local point cloud images, the local point cloud image is plane-divided according to the normal vectors and coordinate positions of the point clouds in the local point cloud image to obtain a plurality of first plane areas included in the non-overlapping common viewing area in the local point cloud image.

3. The method according to claim 1 or 2, characterized in that For the non-overlapping common view area between adjacent partial point cloud images, extracting each planar area belonging to the non-overlapping common view area in at least one partial point cloud image further includes: Acquire local image data of the target scene corresponding to the local point cloud image, the local image data being image data of the same subscene of the target scene as the local point cloud image, synchronously acquired by the second acquisition device when the first acquisition device acquires any local point cloud image of the target scene; Performing element division on the local image data using a preset semantic segmentation algorithm to obtain a plurality of element images in the local image data; Determining a mapping relationship between any one of the local point cloud images and a plurality of element images in the local image data corresponding thereto according to preset calibration parameters of the first acquisition device and the second acquisition device; Based on the mapping relationship, a plurality of second planar areas of the non-overlapping common view area are extracted from any one of the local point cloud images.

4. The method according to claim 2, characterized in that For any of the local point cloud images, plane division is performed on the local point cloud image according to the normal vectors and coordinate positions of the point clouds in the local point cloud image to obtain a plurality of first plane areas included in the non-overlapping common view area in the local point cloud image, including: For any of the local point cloud images, determining the direction of the normal vector of the point cloud of the local point cloud image according to the normal vector distribution of the neighborhood points of the point cloud in the local point cloud image; In the local point cloud image, determining whether the direction angle between the normal vectors of any two point clouds is less than a preset angle threshold, and whether the coordinate position distance between the two point clouds is less than a preset distance threshold; When it is determined that the direction angle between the normal vectors of the two point clouds is less than a preset angle threshold, and the coordinate position distance is less than a preset distance threshold, the two point clouds are divided into the same plane area; The point cloud of the non-overlapping common view area in the local point cloud image is traversed, and the above steps are repeated to obtain a plurality of first planar areas of the non-overlapping common view area in the local point cloud image.

5. The method according to claim 4, characterized in that For any of the local point cloud images, plane division is performed on the local point cloud image according to the normal vectors and coordinate positions of the point clouds in the local point cloud image to obtain a plurality of first plane areas included in the non-overlapping common view area in the local point cloud image, further comprising: In the case where it is determined that the direction angle of the normal vectors of the two point clouds is less than the preset angle threshold and the coordinate position distance is less than the preset distance threshold, after dividing the two point clouds into the same plane area, determining the size of the plane area and the total number of point clouds based on all point cloud data information of any plane area; When the size of the plane area is smaller than a preset size threshold or the total number of point clouds in the plane area is smaller than a preset number threshold, the plane area is marked as an unstable area, and the first plane area does not include the unstable area.

6. The method according to claim 3, characterized in that The step of extracting a plurality of second planar regions of the non-overlapping common view area from any one of the local point cloud images based on the mapping relationship comprises: Based on the mapping relationship, determining point cloud data information of multiple planes corresponding to multiple element images in the local image data from any one of the local point cloud images; determining, based on the classification of the plurality of element images, point cloud data information of a plane of interest from the point cloud data information of the plurality of planes; Based on the point cloud data information of the plane of interest, a plurality of second plane areas of the non-overlapping common view area are extracted from the point cloud data information of the plurality of planes.

7. The method according to claim 6, characterized in that The step of extracting a plurality of second plane regions of the non-overlapping common view area from the point cloud data information of the plurality of planes based on the point cloud data information of the plane of interest comprises: Based on the point cloud data information of the plane of interest, filtering the point cloud data information of non-planes of interest from the local point cloud image to retain the point cloud data information of the plane of interest; A plane fitting process is performed on the retained point cloud data information of the plane of interest using a preset random sampling consistency algorithm to obtain a plurality of second plane areas of the non-overlapping common view area.

8. The method according to claim 1 or 2, characterized in that The performing pose optimization on the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas includes: For any of the local point cloud images, determining the local point cloud images adjacent to the local point cloud image to obtain a local image pair; Based on the extracted planar regions belonging to the non-overlapping common view area, determining, in each pair of partial images, an angle between normal vectors of the planar regions across the partial images; Determining a geometric relationship state between each planar region across the partial image according to an angle between normal vectors of each planar region across the partial image; The multiple local point cloud images are subjected to pose optimization according to the matching relationship between the mutually matching overlapping common view areas and the geometric relationship state between the respective planar areas across the local images.

9. A posture optimization device, characterized in that: The device comprises: a point cloud image acquisition module, configured to acquire a plurality of local point cloud images of a target scene, wherein the local point cloud images include point cloud data information of any local space of the target scene, and the local point cloud images are point cloud data of multiple local spaces obtained by performing trajectory acquisition or point acquisition of multiple points on the target scene by a first acquisition device, and the point cloud data of the multiple local spaces constitute a point cloud data structure of the global scene of the target scene; a common view area determination module, configured to determine whether there is an overlapping common view area between adjacent local point cloud images; a common view region extraction module, configured to determine a matching relationship between overlapping common view regions from adjacent local point cloud images if there are overlapping common view regions between adjacent local point cloud images; a plane region extraction module, configured to extract, for a non-overlapping common view region between adjacent local point cloud images, each plane region belonging to the non-overlapping common view region in at least one local point cloud image; A posture optimization module is used to optimize the postures of the multiple local point cloud images based on the matching relationship between the overlapping common view areas and the extracted planar areas belonging to the non-overlapping common view areas; A three-dimensional point cloud model determination module is used to determine the three-dimensional point cloud model of the target scene based on the multiple local point cloud images after the pose optimization.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images