A method, apparatus, electronic device and storage medium for three-dimensional scene reconstruction

Abstract

The application provides a three-dimensional scene reconstruction method, device, electronic equipment and storage medium, and relates to the technical field of vision. The three-dimensional scene reconstruction method comprises the following steps: acquiring first point cloud data and image data of a current scene; determining a first feature point in the first point cloud data; determining a candidate region according to second point cloud data, wherein the second point cloud data is the first point cloud data excluding the first feature point; identifying a target object according to the candidate region and the image data to obtain a second feature point; and performing three-dimensional scene reconstruction on the current scene based on the first feature point and the second feature point. According to the embodiment of the application, the features of the current scene and the target object can be extracted through the fusion of laser point cloud data and two-dimensional image data, and the accurate positioning of the target object can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vision, in particular to a three-dimensional scene reconstruction method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, the management of warehouses in the logistics industry usually relies on a single sensor to obtain information. Laser sensors can quickly obtain depth information of the environment, have high ranging accuracy, good directivity, and are not easily affected by visible light in the environment. Visual sensors can obtain color and texture information of the scene.

[0003] The present inventors have found that relying solely on laser sensors to obtain information lacks expressiveness of the environment, and relying solely on visual sensors is easily affected by changes in light and lacks expression of depth information, which cannot meet the work requirements of warehouse management in complex scenarios. SUMMARY

[0004] According to an aspect of the present application, a three-dimensional scene reconstruction method is provided, including: obtaining first point cloud data and image data of a current scene; determining a first feature point in the first point cloud data; determining a candidate region according to the first point cloud data; identifying a target object according to the candidate region and the image data to obtain a second feature point; and performing three-dimensional scene reconstruction on the current scene based on the first feature point and the second feature point.

[0005] According to some embodiments, determining the first feature point in the first point cloud data includes: obtaining a normal vector of a plane determined by a first data point and any two neighboring points of the first data point, the first data point being any point in the first point cloud data; determining a plurality of local points in the first point cloud data other than the first data point and any two neighboring points of the first data point based on the first data point and the normal vector; and screening the first feature point from the plurality of local points.

[0006] According to some embodiments, screening the first feature point from the plurality of local points includes: in a case where the number of the plurality of local points is greater than or equal to a first preset threshold, obtaining a first data set and a second data set of the plurality of local points, the second data set being constituted by a projection of the first data set on the plane; calculating an error of the first data set in the process of coordinate conversion with the second data set; and iteratively calculating the error to determine the first feature point.

[0007] According to some embodiments, before obtaining the first point cloud data and the image data of the current scene, the shape feature of a known object, the size feature of the known object, and the color feature of the known object are determined.

[0008] According to some embodiments, determining the candidate region according to the first point cloud data comprises: constructing a plurality of triangular facets by the first point cloud data, wherein each triangular facet is formed by any three adjacent points in the first point cloud data, and each edge of the triangular facet is included in at most two triangular facets; calculating the difference between the normal vectors of any two adjacent triangular facets; determining a plurality of clusters of the first point cloud data according to the calculation result of the difference and a second preset threshold; and determining the cluster as the candidate region if the boundary of the cluster matches the shape feature of the known object.

[0009] According to some embodiments, identifying the target object according to the candidate region and the image data to obtain the second feature point comprises: calculating the actual size of the candidate region to match the size feature of the known object; extracting the color information of the candidate region in the image data to match the color feature of the known object; and determining the second feature point according to the matching result of the candidate region and the size feature of the known object and the color feature of the known object.

[0010] According to some embodiments, performing the three-dimensional scene reconstruction of the current scene based on the first feature point and the second feature point comprises: obtaining second point cloud data according to the first feature point and the second feature point; performing the three-dimensional scene reconstruction of the current scene based on the second point cloud data and the image data; and displaying the result of the three-dimensional scene reconstruction.

[0011] According to an aspect of the present application, a device for three-dimensional scene reconstruction is provided, comprising: a data acquisition module configured to obtain first point cloud data and image data of a current scene, wherein the image data comprises color information; a data processing module configured to obtain a first feature point in the first point cloud data, determine a candidate region according to the first point cloud data, identify a target object according to the candidate region and the image data, and perform three-dimensional scene reconstruction of the current scene based on the first feature point and a second feature point corresponding to the identified target object; and a display module configured to display the result of the three-dimensional scene reconstruction.

[0012] According to some embodiments, the data acquisition module performs time registration on the first point cloud data and the image data.

[0013] According to some embodiments, the first point cloud data of the current scene is acquired, including: the data acquisition module acquires the distance return value, the horizontal scanning angle and the pitching scanning angle of any data point in the current scene; the data acquisition module calculates the three-dimensional space coordinates of the any data point based on the distance return value, the horizontal scanning angle and the pitching scanning angle; and the data acquisition module acquires the first point cloud data according to the three-dimensional space coordinates of the any data point.

[0014] According to an aspect of the present application, a gimbal device is provided, comprising: a vehicle body; a laser range finder, installed on the vehicle body, for acquiring the distance return value, the horizontal scanning angle and the pitching scanning angle as described above during the travel of the vehicle body; a camera, installed on the vehicle body, for acquiring the image data as described above during the travel of the vehicle body; and a driving device for driving the vehicle body to travel.

[0015] According to an aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0016] According to an aspect of the present application, a computer readable storage medium is provided, having a computer program stored thereon, and the program is executed by a processor to implement the method as described above.

[0017] According to the embodiments of the present application, the problem of insufficient stability or low positioning accuracy in the process of three-dimensional scene reconstruction by a single sensor can be solved. The technical solution of the present application fuses the laser depth information and the two-dimensional image data, extracts the features of the current scene environment and objects, and realizes the accurate positioning of the objects in the environment

[0018] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application.

[0020] Figure 1 A flow chart of a method for three-dimensional scene reconstruction according to an example embodiment of the present application is shown.

[0021] Figure 2 A flow chart of scene plane feature extraction according to an example embodiment of the present application is shown.

[0022] Figure 3A flowchart illustrating the determination of candidate regions for a target object in the current scene according to an example embodiment of this application is shown.

[0023] Figure 4 A flowchart illustrating target object recognition in a current scene according to an example embodiment of this application is shown.

[0024] Figure 5 A block diagram of a three-dimensional scene reconstruction apparatus according to an example embodiment of this application is shown.

[0025] Figure 6 A block diagram of an electronic device according to an example embodiment of this application is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0027] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0030] This application provides a method, apparatus, electronic device, and storage medium for three-dimensional scene reconstruction. By fusing laser point cloud data with two-dimensional image data, it can effectively extract feature information in the current scene, improve the positioning accuracy of target objects, and ensure the accuracy of three-dimensional scene reconstruction.

[0031] The following will describe in detail, with reference to the accompanying drawings, a method, apparatus, electronic device, and storage medium for three-dimensional scene reconstruction according to embodiments of this application.

[0032] Figure 1 A flowchart illustrating a method for reconstructing a three-dimensional scene according to an example embodiment of this application is shown.

[0033] like Figure 1 As shown, in step S110, the three-dimensional scene reconstruction device acquires the first point cloud data and image data of the current scene.

[0034] For example, in step S110, the three-dimensional scene reconstruction device performs a full scene scan of the current scene by using a laser rangefinder mounted on the gimbal during the movement of the gimbal, and records the distance return value, horizontal scan angle and pitch scan angle collected by the laser rangefinder at any data point (x1, y1, z1) in the current scene.

[0035] The 3D scene reconstruction device calculates the 3D spatial coordinates of any data point (x1, y1, z1) based on the distance return value, horizontal scanning angle, and pitch scanning angle collected by the laser rangefinder, expressed by the following formula:

[0036] x1 = ρcosαsinβ;

[0037] y1 = ρcosβ;

[0038] z1=ρsinαsinβ.

[0039] Where ρ is the distance return value, α is the horizontal scan angle, and β is the pitch scan angle.

[0040] The 3D scene reconstruction device obtains the first point cloud data of the current scene based on the 3D spatial coordinates of any data point (x1, y1, z1).

[0041] The 3D scene reconstruction device also obtains image data of the current scene through a camera mounted on a gimbal, including color information.

[0042] In step S120, the three-dimensional scene reconstruction device determines the first feature point in the first point cloud data.

[0043] For example, in step S120, the 3D scene reconstruction device arbitrarily selects a first data point P1 from the first point cloud data, and obtains two neighboring points P2 and P3 of the first data point P1. The 3D scene reconstruction device determines a plane π based on P1, P2, and P3, and the normal vector of plane π is denoted as...

[0044] The 3D scene reconstruction device uses the positions and normal vectors of all points in the first point cloud data except for the first data point P1 and its two neighboring points P2 and P3. Determine whether any point other than the first data point P1 and its two neighboring points P2 and P3 is an in-place point.

[0045] If the number of determined inliers is greater than or equal to a first preset threshold, the 3D scene reconstruction device performs local planar fitting on the determined inliers to obtain a first dataset of inliers and a second dataset formed by the projection of the first dataset onto plane π.

[0046] The 3D scene reconstruction device calculates the error between the local points in the first dataset and the corresponding points in the second dataset during the coordinate transformation from the camera coordinate system to the world coordinate system, and performs iterative minimization calculation on the obtained results to obtain the coordinate transformation relationship from the camera coordinate system to the world coordinate system of the first point cloud data.

[0047] The 3D scene reconstruction device removes the local points in the first dataset that have already undergone error calculation from the first point cloud data, and continues to determine the local points of the remaining points in the first point cloud data and calculate the errors in the coordinate transformation process from the camera coordinate system to the world coordinate system for the local points in the first dataset and the points in the second dataset, until all points in the first point cloud data have been calculated.

[0048] The 3D scene reconstruction device determines the local points in the first point cloud data in the camera coordinate system as the first feature points in the world coordinate system based on the coordinate transformation relationship from the camera coordinate system to the world coordinate system.

[0049] In step S130, the three-dimensional scene reconstruction device determines the candidate region based on the first point cloud data.

[0050] For example, in step S130, the three-dimensional scene reconstruction device constructs triangular patches using any three neighboring points in the first point cloud data, thereby forming multiple triangular patches, and each edge of each triangular patch is contained in at most two triangular patches.

[0051] The 3D scene reconstruction device obtains the normal vector of a triangular facet based on the vector between the three points that make up the triangular facet, and calculates the difference between the normal vectors of any two adjacent triangular facets.

[0052] The 3D scene reconstruction device compares the calculated difference between the normal vectors of any two adjacent triangular faces with a second preset threshold. If the difference meets the second preset threshold, the 3D scene reconstruction device determines that the vertices of the two triangular faces belong to the same cluster. After dividing the points in the first point cloud data into multiple different clusters, the 3D scene reconstruction device obtains the boundary of each cluster (i.e., the outline of a possible target object) using an image edge detection algorithm and matches it with the shape features of pre-obtained known objects. If a match is successful, the 3D scene reconstruction device identifies the cluster that matches the shape features of the known objects as a candidate region.

[0053] According to some embodiments, before acquiring the first point cloud data and image data of the current scene, the 3D scene reconstruction device extracts features of known objects through preset algorithms and models to obtain a feature database of known objects, including the shape features, size features and color features of known objects.

[0054] In step S140, the three-dimensional scene reconstruction device identifies the target object based on the candidate region and image data to obtain the second feature point.

[0055] For example, in step S140, the 3D scene reconstruction device calculates the actual size corresponding to the determined candidate region through a coordinate transformation model, and matches the actual size of the candidate region with the size features of the known object.

[0056] The 3D scene reconstruction device extracts the color information of candidate regions from the image data and matches the color information of the candidate regions with the color features of known objects.

[0057] When the candidate region matches the size and color features of the known object, the 3D scene reconstruction device obtains the second feature points and two-dimensional image corresponding to the candidate region, thereby realizing the recognition of the target object.

[0058] In step S150, the three-dimensional scene reconstruction device reconstructs the current scene based on the first feature point and the second feature point.

[0059] For example, in step S150, the 3D scene reconstruction device performs coordinate transformation on the second feature point and superimposes it with the first feature point to obtain the second point cloud data in the world coordinate system.

[0060] The 3D scene reconstruction device reconstructs the current scene based on the color information in the second point cloud data and image data, and displays the 3D scene reconstruction result of the current scene through a visualization interface.

[0061] According to the embodiments of this application, the technical solution of this application fuses the depth image data obtained by a lidar device (such as a laser rangefinder) with the two-dimensional image data obtained by a vision device (such as a camera), extracts the planar features of the current scene, identifies the target objects in the current scene, and uses this to reconstruct the three-dimensional scene, thereby achieving accurate positioning of objects in the scene and ensuring the stability and real-time performance of the three-dimensional scene reconstruction.

[0062] Figure 2 A flowchart illustrating scene planar feature extraction according to an example embodiment of this application is shown.

[0063] like Figure 2 As shown, in step S210, the three-dimensional scene reconstruction device acquires the normal vector of the plane determined by the first data point and any two neighboring points of the first data point.

[0064] For example, in step S210, the 3D scene reconstruction device can obtain the local points in the first point cloud data through the Random Sample Consensus (RANSAC) method.

[0065] The 3D scene reconstruction device arbitrarily selects a first data point P1 from the first point cloud data, and obtains two neighboring points P2 and P3 of the first data point P1. Based on P1, P2, and P3, the 3D scene reconstruction device determines a plane π, and the normal vector of plane π is denoted as...

[0066] In step S220, the three-dimensional scene reconstruction device determines multiple local points of the first point cloud data based on the first data point and the normal vector.

[0067] For example, in step S220, for points X in the first point cloud data other than the first data point P1 and two neighboring points P2 and P3, the 3D scene reconstruction device determines whether point X belongs to an in-place point based on the position of point X and the following formula:

[0068]

[0069] Where (X-P1) represents the difference in coordinate vectors between point X and point P1, excluding P1, P2, and P3. The normal vector of plane π is determined based on P1, P2, and P3. ε is a preset parameter that can be adjusted according to the actual needs of the scenario.

[0070] In step S230, when the number of multiple in-place points is greater than or equal to a first preset threshold, the 3D scene reconstruction device acquires a first dataset and a second dataset of multiple in-place points.

[0071] For example, in step S230, if the number of local points determined by the 3D scene reconstruction device is greater than or equal to the first preset threshold, the 3D scene reconstruction device may use the iterative nearest point method to perform local planar fitting on the determined local points.

[0072] The 3D scene reconstruction device forms a first dataset M based on the determined local points, and a second dataset D based on the projection of the first dataset M onto plane π. The 3D scene reconstruction device performs a pre-defined algebraic iterative calculation to minimize the error between the local points in the first dataset and the corresponding points in the second dataset during the coordinate transformation from the camera coordinate system to the world coordinate system, in order to find the closest point pair (m) between the first dataset M and the second dataset D. i Then, the optimal matching parameters R and t (i.e., the coordinate transformation relationship from the camera coordinate system to the world coordinate system) are calculated:

[0073]

[0074] Where R represents rotation, t represents translation, and m i Let dj be a point in the first dataset M, dj be a point in the second dataset D, n be the number of inliers, and ω be the number of points in the dataset D. i,j The preset weighting coefficients can be adjusted according to the actual needs of the scenario.

[0075] In step S240, the three-dimensional scene reconstruction device determines the first feature point.

[0076] For example, in step S240, the 3D scene reconstruction device removes the local points in the first dataset that have already undergone error calculation, and continues to perform the operations of steps S210 to S230 on the remaining points in the first point cloud until the data of all points in the first point cloud data has been calculated.

[0077] The 3D scene reconstruction device determines the local points in the first point cloud data in the camera coordinate system as the first feature points in the world coordinate system based on the coordinate transformation relationship from the camera coordinate system to the world coordinate system.

[0078] According to the embodiments of this application, the technical solution of this application extracts the planar frame features of the current scene from the first point cloud data by combining the RANSAC method and the iterative nearest point method, so as to extract other features from the remaining point cloud data in the first point cloud and achieve accurate positioning of the target object.

[0079] Figure 3 A flowchart illustrating the determination of candidate regions for a target object in the current scene according to an example embodiment of this application is shown.

[0080] like Figure 3As shown, in step S310, the three-dimensional scene reconstruction device constructs multiple triangular patches using the first point cloud data.

[0081] For example, in step S310, the 3D scene reconstruction device connects any three adjacent points in the space of the first point cloud data to construct a triangular patch. The 3D scene reconstruction device generates multiple triangular patches based on the second point cloud data, wherein each edge of each triangular patch is contained in at most two triangular patches.

[0082] In step S320, the three-dimensional scene reconstruction device calculates the difference between the normal vectors of any two adjacent triangular facets.

[0083] For example, in step S320, the 3D scene reconstruction device selects three neighboring points P in space. i，j P i+1，j and P i，j+1 This forms a triangular facet.

[0084] 3D scene reconstruction device calculation point P i，j and point P i+1，j Vectors between And point P i，j and P i，j+1 Vectors between

[0085] The 3D scene reconstruction device calculates the normal vector of a triangular facet according to the following formula.

[0086]

[0087] in, Triangular facet and normal vector The coordinates of the intersection point are given by T, which is its transpose.

[0088] For any two adjacent triangular facets, the normal vector and The difference is calculated by the 3D scene reconstruction device using the following formula:

[0089]

[0090] In step S330, the three-dimensional scene reconstruction device determines multiple clusters of the first point cloud data based on the calculation results of the difference.

[0091] For example, in step S330, the 3D scene reconstruction device compares the calculated difference between the normals of any two adjacent triangular faces with a second preset threshold. If the calculated difference meets the second preset threshold, the 3D scene reconstruction device determines that the vertices of the two triangular faces belong to the same cluster, and thereby divides the points in the first point cloud data into multiple different clusters.

[0092] The 3D scene reconstruction device obtains the boundary of each cluster based on the image edge detection algorithm.

[0093] In step S340, if the boundary of the cluster matches the shape features of a known object, the 3D scene reconstruction device identifies the cluster as a candidate region.

[0094] For example, in step S340, based on the matching results of the boundaries of the cluster (i.e. the outline of the possible target object) and the shape features of the known objects obtained in advance, the three-dimensional scene reconstruction device determines the clusters that match the shape features of the known objects as candidate regions of the target objects.

[0095] According to the embodiments of this application, the technical solution of this application will represent the collected depth image information in the form of a triangular mesh, and extract the corresponding shape feature information from it to facilitate the identification of target objects.

[0096] Figure 4 A flowchart illustrating target object recognition in a current scene according to an example embodiment of this application is shown.

[0097] like Figure 4 As shown, in step S410, the 3D scene reconstruction device calculates the actual size of the candidate region to match the size features of the known object.

[0098] For example, in step S410, the 3D scene reconstruction device can calculate the actual dimensions of the determined candidate region, such as length, width, and height, through a homogeneous coordinate transformation system model. The 3D scene reconstruction device then matches the actual dimensions of the candidate region with the pre-obtained dimensional features of known objects.

[0099] In step S420, the three-dimensional scene reconstruction device extracts the color information corresponding to the candidate region in the image data to match the color features of the known object.

[0100] For example, in step S420, the 3D scene reconstruction device can extract the color information corresponding to the determined candidate region in the image data, such as color histogram and color moments, through coordinate transformation between the laser and the camera (e.g., external calibration parameters between the laser and the camera). The 3D scene reconstruction device then matches the color information of the candidate region with the color features of the known objects obtained in advance.

[0101] In step S430, based on the matching results between the candidate region and the size and color features of the known object, the 3D scene reconstruction device determines whether the target object is identifiable.

[0102] For example, in step S430, the three-dimensional scene reconstruction device obtains the actual size and color information of each candidate region, and matches them with the size and color features of the known objects obtained in advance.

[0103] The 3D scene reconstruction device obtains the second feature points and two-dimensional image corresponding to the candidate region based on the matching results of the size and color features of the candidate region and the known object, thereby realizing the recognition of the target object.

[0104] According to the embodiments of this application, the technical solution of this application can achieve dual matching of the size and color information of the target object, thereby achieving accurate separation of the target object in the current scene.

[0105] Figure 5 A block diagram of a three-dimensional scene reconstruction apparatus according to an example embodiment of this application is shown.

[0106] like Figure 5 As shown, the 3D scene reconstruction device 500 includes a data acquisition module 510, a data processing module 520, and a display module 530.

[0107] The data acquisition module 510 can acquire the first point cloud data and image data of the current scene through a laser rangefinder and a camera, respectively.

[0108] According to some embodiments, the data acquisition module 510 can obtain the two-dimensional distance return value, horizontal scanning angle and pitch scanning angle of any data point in the current scene through a laser rangefinder set on the gimbal during the movement of the gimbal body. Furthermore, the data acquisition parameters of the laser rangefinder are preset. For example, the acquisition frame rate of the laser rangefinder can be set to 18 frames / second, the scanning range can be set to 180 degrees, and the scanning interval angle can be set to 0.5 degrees.

[0109] The data acquisition module 510 calculates the three-dimensional spatial coordinates of any data point based on the two-dimensional distance return value, horizontal scanning angle, and pitch scanning angle to form the first point cloud data.

[0110] The data acquisition module 510 can also obtain current scene image data during the movement of the gimbal body through a camera set on the gimbal. This includes color information of the current scene and objects. Furthermore, the data acquisition parameters of the camera are preset, for example, the image acquisition frame rate is set to 5 frames / second and the image size is set to 1920*1080 pixels.

[0111] According to some embodiments, when the data acquisition frequencies of the laser rangefinder and the camera are not synchronized, the data acquisition module 510 can establish a correspondence between the data acquired by the laser rangefinder and the camera through asynchronous data fusion, so as to achieve registration of the acquisition time of the first point cloud data and the image data.

[0112] The data processing module 520 first determines the local points in the first point cloud data, and obtains the coordinate transformation relationship from the camera coordinate system to the world coordinate system of the first point cloud data by the error in the coordinate transformation process from the camera coordinate system to the world coordinate system between the first dataset of local points and the second dataset formed by the planar projection of the first dataset, so as to determine the planar feature points (i.e. the first feature points) of the current scene.

[0113] The data processing module 520 constructs multiple triangular patches based on the first point cloud data and calculates the difference between the normal vectors of any two adjacent triangular patches to determine whether multiple clusters in the first point cloud data are candidate regions.

[0114] The data processing module 520 determines whether the target object is identifiable based on the matching results between the actual size and color information of the candidate region and the size and color features of the known objects obtained in advance, and generates second point cloud data based on the second feature points and the first feature points corresponding to the identified target object.

[0115] The data processing module 520 performs 3D scene reconstruction of the current scene based on the second point cloud data and image data.

[0116] Display module 530 is used to display the results of the 3D scene reconstruction of the current scene through a visual interface.

[0117] Figure 6 A block diagram of an electronic device according to an example embodiment of this application is shown.

[0118] like Figure 6 As shown, the electronic device 600 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0119] like Figure 6 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the methods described in this specification according to the various exemplary embodiments of this application. For example, the processing unit 610 can perform, for example... Figure 1 The method shown.

[0120] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0121] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0122] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0123] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0124] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0125] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0126] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0127] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0128] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0129] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0130] According to some embodiments of this application, the technical solution of this application, combined with data collected by laser equipment and vision equipment, improves the accuracy and efficiency of object recognition in complex scenes and ensures the stability and real-time performance of 3D scene reconstruction.

[0131] The embodiments of this application have been described in detail above. These descriptions are solely for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, its specific implementation methods, and its application scope, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of three-dimensional scene reconstruction, characterized in that, The method comprises: acquiring first point cloud data and image data of a current scene; determining first feature points in the first point cloud data; determining a candidate region according to the first point cloud data; identifying a target object according to the candidate region and the image data to obtain second feature points; reconstructing a three-dimensional scene of the current scene based on the first feature points and the second feature points; wherein, before acquiring the first point cloud data and the image data of the current scene, shape features, size features and color features of a known object are determined; wherein, determining the candidate region according to the first point cloud data comprises: constructing a plurality of triangular facets from the first point cloud data, wherein each triangular facet is composed of any three adjacent points in the first point cloud data, and each edge of the triangular facet is included in at most two triangular facets; calculating the difference between the normal vectors of any two adjacent triangular facets; determining a plurality of clusters of the first point cloud data according to the calculation result of the difference and a second preset threshold; determining the cluster as the candidate region if the boundary of the cluster matches the shape features of the known object; wherein, identifying the target object according to the candidate region and the image data to obtain the second feature points comprises: calculating the actual size of the candidate region to match the size features of the known object; extracting the color information of the candidate region in the image data to match the color features of the known object; determining the second feature points according to the matching result of the candidate region, the size features of the known object and the color features of the known object; wherein, reconstructing the three-dimensional scene of the current scene based on the first feature points and the second feature points comprises: acquiring second point cloud data according to the first feature points and the second feature points; reconstructing the three-dimensional scene of the current scene based on the second point cloud data and the image data; displaying the result of the three-dimensional scene reconstruction.

2. The method of claim 1, wherein, Determining the first feature points in the first point cloud data comprises: acquiring a normal vector of a plane determined by a first data point and any two adjacent points of the first data point, the first data point being any point in the first point cloud data; determining a plurality of local points in the first point cloud data other than the first data point and any two adjacent points of the first data point according to the first data point and the normal vector; screening the first feature points from the plurality of local points.

3. The method of claim 2, wherein, Screening the first feature points from the plurality of local points comprises: if the number of the plurality of local points is greater than or equal to a first preset threshold, acquiring a first data set and a second data set of the plurality of local points, the second data set being composed of the projection of the first data set on the plane; calculating the error of the first data set in the process of coordinate conversion with the second data set; iteratively calculating the error to determine the first feature points.

4. An apparatus for three-dimensional scene reconstruction, the apparatus comprising: The method comprises: a data acquisition module, configured to acquire first point cloud data and image data of a current scene, the image data comprising color information; a data processing module, configured to acquire first feature points in the first point cloud data; determine a candidate region according to the first point cloud data; and identify a target object according to the candidate region and the image data; perform three-dimensional scene reconstruction of the current scene based on the first feature points and second feature points corresponding to the identified target object; a display module, configured to display a result of the three-dimensional scene reconstruction; wherein the data processing module is further configured to determine shape features of a known object, size features of the known object, and color features of the known object before acquiring the first point cloud data and the image data of the current scene; and construct a plurality of triangular facets from the first point cloud data, wherein each triangular facet is composed of any three adjacent points in the first point cloud data, and each edge of the triangular facet is included in at most two triangular facets; calculate a difference degree between normal vectors of any two adjacent triangular facets; determine a plurality of clusters of the first point cloud data according to a calculation result of the difference degree and a second preset threshold; determine the cluster as the candidate region in a case where a boundary of the cluster matches the shape features of the known object; and calculate an actual size of the candidate region to match the size features of the known object; extract color information corresponding to the candidate region in the image data to match the color features of the known object; determine the second feature points according to a matching result of the candidate region and the size features of the known object and the color features of the known object; and acquire second point cloud data according to the first feature points and the second feature points; perform the three-dimensional scene reconstruction of the current scene based on the second point cloud data and the image data.

5. The apparatus of claim 4, wherein, The data acquisition module performs acquisition time registration on the first point cloud data and the image data.

6. The apparatus of claim 4, wherein, Acquiring first point cloud data of a current scene comprises: the data acquisition module acquires a distance return value, a horizontal scanning angle, and a pitch scanning angle of any data point in the current scene; the data acquisition module calculates a three-dimensional space coordinate of the any data point based on the distance return value of the any data point, the horizontal scanning angle of the any data point, and the pitch scanning angle of the any data point; the data acquisition module acquires the first point cloud data according to the three-dimensional space coordinate of the any data point.

7. A gimbal device, comprising: comprise: a vehicle body; a laser range finder, mounted on the vehicle body, configured to acquire the distance return value, the horizontal scanning angle, and the pitch scanning angle of any data point in the current scene during travel of the vehicle body; a camera, mounted on the vehicle body, configured to acquire the image data in claim 4 or 5 during travel of the vehicle body; a driving device, configured to drive the vehicle body to travel.

8. An electronic device, comprising: comprise: one or more processors; a storage device, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a method as claimed in any of claims 1-3.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, which when executed by a processor, implements a method as claimed in any of claims 1-3.

Images