Key frame image acquisition method, device, system and three-dimensional reconstruction method

Abstract

The present disclosure relates to a key frame image acquisition method, device, system and three-dimensional reconstruction method, and relates to the field of computer technology. The acquisition method includes: determining the point cloud data of the target object according to the multi-frame depth images of the target object acquired by the image acquisition device at the current shooting position and the internal parameters of the image acquisition device; establishing a three-dimensional model that can surround the target object according to the position information of the point cloud data of the target object, the internal parameters of the image acquisition device and the external parameters at the current shooting position, and the surface of the three-dimensional model is divided into multiple grids, each grid corresponding to a shooting position; determining the key frame image of the target object at the corresponding shooting position in the multi-frame depth images of each shooting position according to the grids corresponding to each shooting position, the shooting position and shooting direction of the multi-frame depth images acquired at each shooting position.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a key frame image acquisition method, a key frame image acquisition device, a key frame image acquisition system, a key frame image three-dimensional reconstruction method, a key frame image three-dimensional reconstruction device and a non-volatile computer-readable storage medium. Background Art

[0002] 3D reconstruction technology is a research hotspot in the field of computer graphics and computer vision. Before the emergence of consumer-grade depth cameras, the input data for 3D reconstruction usually only consisted of RGB (Red Green Blue) images. Based on a series of RGB images taken at different angles, the 3D model of the object is restored through a stereo vision algorithm. These RGB images are the key frames that 3D reconstruction relies on. In order to accurately and completely reconstruct the 3D model of the object, the key frames must cover all angles and cannot be too blurry.

[0003] With the emergence of various consumer-grade depth cameras, 3D reconstruction technology based on depth cameras has developed rapidly. With the help of depth images collected by depth cameras, the 3D model of an object can be quickly restored. In order to restore the realism of the 3D model, it is also necessary to restore the real texture of the object through texture mapping. Texture mapping relies on a series of key frames covering all angles of the 3D model.

[0004] Therefore, no matter which 3D reconstruction technology is used, in order to obtain a high-quality 3D model, key frames that cover comprehensive angles and are clear enough are essential.

[0005] In the related art, time sampling can be performed according to the time interval and blur degree of each frame image to select key frame images at each angle; or multiple cameras can be set up around the target object to obtain key frame images at each angle. Summary of the invention

[0006] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problems: the key frame images do not cover all shooting angles, resulting in a decrease in the quality of key frame image acquisition; or the cost of obtaining key frame images of complete angles is too high.

[0007] In view of this, the present disclosure proposes a technical solution for capturing key frame images, which can capture key frame images of complete shooting angles without increasing costs, thereby improving the capture quality of key frame images.

[0008] According to some embodiments of the present disclosure, a method for acquiring key frame images is provided, including: determining point cloud data of the target object based on a multi-frame depth image of the target object acquired by an image acquisition device at a current shooting position and internal parameters of the image acquisition device; establishing a three-dimensional model that can surround the target object based on position information of the point cloud data of the target object, the internal parameters of the image acquisition device and external parameters at the current shooting position, wherein the surface of the three-dimensional model is divided into multiple grids, each grid corresponding to a shooting position; determining the key frame image of the target object at the corresponding shooting position in the multi-frame depth image of each shooting position based on the grid corresponding to each shooting position and the shooting position and shooting direction of the multi-frame depth images acquired at each shooting position.

[0009] In some embodiments, establishing a three-dimensional model that can surround a target object includes: calculating a projection matrix of the three-dimensional model from a three-dimensional coordinate system to a two-dimensional coordinate system based on internal parameters of an image acquisition device; and establishing a three-dimensional model based on the projection matrix, position information of vertices of each grid, and external parameters of the image acquisition device at a shooting position corresponding to each grid.

[0010] In some embodiments, establishing a three-dimensional model that can surround the target object includes: performing plane fitting based on point cloud data of the target object to determine the plane where the target object is located; and establishing the three-dimensional model based on the position of the target object on the plane.

[0011] In some embodiments, establishing a three-dimensional model according to the position of the target object on the plane model includes: determining the center position of a circle according to the position of the target object on the plane model; and establishing a three-dimensional hemispherical model as the three-dimensional model on the plane model according to the center position of the circle.

[0012] In some embodiments, determining the key frame image of the target object at the corresponding shooting position in the multi-frame depth image of each shooting position includes: determining the key frame image of the current shooting position according to the first ray, the second ray, and the grid position corresponding to the current shooting position of the multi-frame depth image of the current shooting position, the second ray is a ray that takes the position of the image acquisition device in the world coordinate system as the starting point and points to the center point of the three-dimensional model in the plane where the target object is located, and the first ray points to the shooting direction of the image acquisition device at the current shooting position.

[0013] In some embodiments, determining the key frame image of the current shooting position based on the first ray, the second ray of the multiple-frame depth images of the current shooting position, and the grid position corresponding to the current shooting position includes: when the first ray and the second ray of a frame depth image of the current shooting position both pass through the grid corresponding to the current shooting position, determining the frame depth image as the candidate image of the current shooting position; and determining the key frame image of the current shooting position in each candidate image based on the clarity of each candidate image.

[0014] In some embodiments, according to the clarity of each candidate image, determining the key frame image of the current shooting position in each candidate image includes: using an edge detection operator to calculate edge feature information of each candidate image; and determining the clarity of each candidate image according to statistical characteristics of each edge feature information.

[0015] In some embodiments, determining the key frame image of the target object at the corresponding shooting position in the multi-frame depth image of each shooting position includes: in the case where there are candidate images with a clarity greater than a threshold, determining the corresponding candidate images as key frame images, taking the next shooting position of the image acquisition device as the current shooting position, and determining the corresponding key frame image in the multi-frame depth image at the current shooting position; in the case where there are no candidate images with a clarity greater than the threshold, acquiring multiple-frame depth images again at the current shooting position until there is a candidate image with a clarity greater than the threshold.

[0016] In some embodiments, the position and shooting direction of the image acquisition device in the world coordinate system are calculated according to external parameters of the image acquisition device at the current shooting position.

[0017] In some embodiments, the acquisition method also includes: in response to the image acquisition device acquiring multiple frames of depth images at the current shooting position, performing a first marking process on the corresponding grid of the current shooting position on the three-dimensional model; in response to determining the key frame image of the current shooting position, performing a second marking process on the corresponding grid of the current shooting position on the three-dimensional model to indicate that the key frame image of the current shooting position has been acquired.

[0018] In some embodiments, the acquisition method further includes: performing three-dimensional reconstruction of the target object based on the key frame images of the target object at each shooting position.

[0019] According to other embodiments of the present disclosure, a method for three-dimensional reconstruction of key frame images is provided, comprising: utilizing the key frame image acquisition method of any one of the above embodiments to acquire key frame images of a target object at each shooting position; and performing three-dimensional reconstruction of the target object based on the key frame images of the target object at each shooting position.

[0020] According to some other embodiments of the present disclosure, a key frame image acquisition device is provided, including: a point cloud determination unit, used to determine point cloud data of a target object based on a multi-frame depth image of the target object acquired by an image acquisition device at a current shooting position and internal parameters of the image acquisition device; a building unit, used to build a three-dimensional model that can surround the target object based on position information of the point cloud data of the target object, internal parameters of the image acquisition device and external parameters at the current shooting position, wherein the surface of the three-dimensional model is divided into multiple grids, each grid corresponding to a shooting position; a key frame determination unit, used to determine the key frame image of the target object at a corresponding shooting position in the multi-frame depth image of each shooting position based on the grid corresponding to each shooting position, the shooting position and shooting direction of the multi-frame depth images acquired at each shooting position.

[0021] In some embodiments, the establishing unit calculates the projection matrix of the three-dimensional model from the three-dimensional coordinate system to the two-dimensional coordinate system based on the internal parameters of the image acquisition device; and establishes the three-dimensional model based on the projection matrix, the position information of the vertices of each grid, and the external parameters of the image acquisition device at the shooting position corresponding to each grid.

[0022] In some embodiments, the building unit performs plane fitting based on the point cloud data of the target object to determine the plane where the target object is located; and builds a three-dimensional model based on the position of the target object on the plane.

[0023] In some embodiments, the establishing unit determines the position of the center of the circle according to the position of the target object on the plane model; and establishes a three-dimensional hemispherical model on the plane model as the three-dimensional model according to the position of the center of the circle.

[0024] In some embodiments, the key frame determination unit determines the key frame image of the current shooting position based on the first ray, the second ray, and the grid position corresponding to the current shooting position of the multi-frame depth image of the current shooting position, the second ray is a ray that takes the position of the image acquisition device in the world coordinate system as the starting point and points to the center point of the three-dimensional model in the plane where the target object is located, and the first ray points to the shooting direction of the image acquisition device at the current shooting position.

[0025] In some embodiments, when the first ray and the second ray of a frame depth image of the current shooting position both pass through the grid corresponding to the current shooting position, the key frame determination unit determines the frame depth image as a candidate image of the current shooting position; and determines the key frame image of the current shooting position among the candidate images according to the clarity of each candidate image.

[0026] In some embodiments, the key frame determination unit calculates edge feature information of each candidate image using an edge detection operator; and determines the clarity of each candidate image based on statistical features of each edge feature information.

[0027] In some embodiments, when there are candidate images with clarity greater than a threshold, the key frame determination unit determines the corresponding candidate images as key frame images, takes the next shooting position of the image acquisition device as the current shooting position, and determines the corresponding key frame images in the multi-frame depth images at the current shooting position; when there are no candidate images with clarity greater than the threshold, the multi-frame depth images are acquired again at the current shooting position until there is a candidate image with clarity greater than the threshold.

[0028] In some embodiments, the position and shooting direction of the image acquisition device in the world coordinate system are calculated according to external parameters of the image acquisition device at the current shooting position.

[0029] In some embodiments, the acquisition device also includes: a marking unit, which is used to perform a first marking process on the corresponding grid of the current shooting position on the three-dimensional model in response to the image acquisition device acquiring multiple frames of depth images at the current shooting position; in response to determining the key frame image of the current shooting position, perform a second marking process on the corresponding grid of the current shooting position on the three-dimensional model to indicate that the key frame image of the current shooting position has been acquired.

[0030] In some embodiments, the acquisition device further includes: a reconstruction unit, configured to perform three-dimensional reconstruction of the target object based on key frame images of the target object at each shooting position.

[0031] According to some further embodiments of the present disclosure, a three-dimensional reconstruction device of key frame images is provided, comprising: an acquisition unit, used to execute the key frame image acquisition method of any of the above embodiments, to acquire key frame images of the target object at each shooting position; and a reconstruction unit, used to perform three-dimensional reconstruction of the target object based on the key frame images of the target object at each shooting position.

[0032] According to some further embodiments of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the key frame image acquisition method or the key frame image three-dimensional reconstruction method in any one of the above-mentioned embodiments based on instructions stored in the memory device.

[0033] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for acquiring key frame images or the method for three-dimensional reconstruction of key frame images in any of the above-mentioned embodiments is implemented.

[0034] According to some further embodiments of the present disclosure, a key frame image acquisition system is provided, comprising: a key frame image acquisition device, used to execute the key frame image acquisition method in any of the above embodiments; and an image acquisition device, used to acquire multiple frames of depth images of a target object at different shooting positions.

[0035] In the above embodiment, a three-dimensional model that can surround the target object is established, and the key frame image acquisition is guided according to the grid corresponding to each shooting position on the surface of the three-dimensional model. In this way, the key frame image acquisition direction can be guided synchronously when scanning the object, so that the key frame acquired can completely cover each shooting angle, thereby improving the acquisition quality of the key frame image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0037] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings:

[0038] Figure 1 A flowchart showing some embodiments of the key frame image acquisition method disclosed in the present invention;

[0039] Figure 2 Show Figure 1 Flowcharts of some embodiments of step 130;

[0040] Figure 3 Schematic diagrams showing some embodiments of the key frame image acquisition method disclosed in the present invention;

[0041] Figure 4 A flowchart showing some embodiments of the method for 3D reconstruction of key frame images disclosed herein;

[0042] Figure 5 A block diagram showing some embodiments of a key frame image acquisition device of the present disclosure;

[0043] Figure 6 A block diagram showing some embodiments of a three-dimensional reconstruction apparatus of a key frame image disclosed in the present invention;

[0044] Figure 7 A block diagram showing some embodiments of the electronic device of the present disclosure;

[0045] Figure 8 A block diagram showing some other embodiments of the electronic device of the present disclosure;

[0046] Fig. 9 A block diagram showing some embodiments of a key frame image acquisition system of the present disclosure. DETAILED DESCRIPTION

[0047] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0048] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0049] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0050] Technologies, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies, methods and equipment should be considered part of the authorization specification.

[0051] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0052] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0053] In order to solve the above technical problems, in an embodiment of the present disclosure, a user can hold a mobile phone or tablet computer equipped with a depth camera and scan around the target object, and collect a certain number of depth images at each shooting position.

[0054] In this way, the collected depth images can cover all angles of the object, and a sufficiently clear depth image can be selected as a key frame image within a certain shooting angle range, thereby reducing data redundancy.

[0055] Moreover, during the acquisition process, key frame images can be determined synchronously from the depth images obtained from each shooting position, thereby improving the efficiency of key frame image acquisition; it also provides users with visual feedback so that users can instantly know whether the key frame images have been acquired at the corresponding shooting position and angle, thereby ensuring that the key frame images can cover the complete angle range.

[0056] In addition, the technical solution of the present disclosure does not require the installation of a camera array, thus reducing hardware costs. For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0057] Figure 1 A flow chart showing some embodiments of the method for collecting key frame images disclosed in the present invention.

[0058] like Figure 1 As shown, the method includes: step 110, determining point cloud data of the target object; step 120, establishing a three-dimensional model surrounding the target object; and step 130, determining a key frame image.

[0059] In step 110, point cloud data of the target object is determined based on the multiple depth images of the target object acquired by the image acquisition device at the current shooting position and the internal parameters of the image acquisition device. For example, multiple depth images may be acquired at the current shooting position, or the orientation of the image acquisition device may be adjusted to acquire multiple depth images from different shooting directions.

[0060] In some embodiments, the object to be scanned, i.e., the target object, can be placed on a suitable plane (such as a desktop, the ground, etc.), and the user holds a mobile phone or tablet computer equipped with a depth camera and aims it at the target object to collect corresponding RGB-D (Red Green Blue-Deep) data; based on the collected RGB-D data, the point cloud data required for three-dimensional reconstruction is calculated.

[0061] For example, if the image acquisition device is a depth camera, the internal parameter matrix is:

[0062]

[0063] f x and f y is the focal length of the depth camera, c x and c y is the principal point coordinate of the depth camera. The internal parameter matrix does not change with the movement of the depth camera.

[0064] The depth value of a point (u, v) on a certain frame of depth image is d, and the point cloud coordinates corresponding to the point can be determined as:

[0065]

[0066] In step 120, a three-dimensional model that can surround the target object is established based on the position information of the point cloud data of the target object, the internal parameters of the image acquisition device, and the external parameters at the current shooting position. The surface of the three-dimensional model is divided into multiple grids, each grid corresponding to a shooting position.

[0067] In some embodiments, after obtaining the point cloud data of each frame of the depth image at the current shooting position, the posture matrix (ie, camera external parameters) corresponding to each frame of the depth image can be solved by algorithms such as ICP (Iterative Closest Point):

[0068]

[0069] That is, the camera external parameters are composed of the rotation matrix R and the translation column vector T:

[0070]

[0071] Camera extrinsic parameters are related to the shooting position.

[0072] In some embodiments, a plane fitting is performed based on the point cloud data of the target object to determine the plane where the target object is located; and a three-dimensional model is established based on the position of the target object on the plane. For example, the position of the center of a circle is determined based on the position of the target object on the plane model; and a three-dimensional hemispherical model is established on the plane model as the three-dimensional model based on the position of the center of the circle.

[0073] For example, after obtaining enough point cloud data, algorithms such as RANSAC (RANdom SAmple Consensus) can be used to perform plane fitting processing.

[0074] First, three points can be randomly selected from the point cloud data, and a candidate plane can be determined based on these three points; then, the distances from all other points to the candidate plane can be calculated; if the distance is less than the distance threshold, the corresponding point is considered to be in the candidate plane and marked as an inlier; otherwise, it is marked as an outlier; if the number of inliers exceeds the number threshold, the candidate plane is saved as a plane model; if the number of inliers does not exceed the number threshold, three points are reselected to determine the candidate plane.

[0075] For example, after obtaining the plane where the target object is placed, the three-dimensional hemispherical model can be rendered to the corresponding position on the plane (such as using openGL for efficient rendering), so that the three-dimensional hemispherical model can cover the target object.

[0076] In order to ensure that the rendered three-dimensional hemispherical model is consistent with the physical space (ie, the world coordinate system), that is, to ensure that the position of the three-dimensional hemispherical model in the world coordinate system does not change with the camera position, the real-time projection matrix and posture matrix of the current shooting position can be used to establish the three-dimensional hemispherical model.

[0077] In some embodiments, the projection matrix of the three-dimensional model from the three-dimensional coordinate system to the two-dimensional coordinate system is calculated based on the internal parameters of the image acquisition device; the three-dimensional model is established based on the projection matrix, the position information of the vertices of each grid, and the external parameters of the image acquisition device at the shooting position corresponding to each grid.

[0078] For example, according to the internal parameter matrix of the depth camera, calculate the projection matrix P corresponding to the current shooting position:

[0079]

[0080] width and height are the width and height of the depth image respectively, far is the current far plane of the depth camera, and near is the current near plane.

[0081] Multiple groups of vertices can be set on the surface of the 3D model, and each group of vertices can determine a grid. The coordinates of a group of vertices are M, and gl_Position = P × V × M when rendering. According to the gl_Position values ​​corresponding to each group of vertices, a 3D model with the surface divided into multiple grids can be established. These grids corresponding to different shooting positions are part of the 3D model UI (User Interface), which can be used as a basis for screening key frame images, and can also guide the acquisition of key frame images covering the full angle range.

[0082] In step 130 , according to the grids corresponding to each shooting position, the shooting position and shooting direction of the multi-frame depth images acquired at each shooting position, the key frame image of the target object at the corresponding shooting position is determined in the multi-frame depth images at each shooting position.

[0083] In some embodiments, the Figure 2 The embodiment in determines the key frame image.

[0084] Figure 2 Show Figure 1 Flowchart of some embodiments of step 130 in FIG.

[0085] like Figure 2 As shown, step 130 includes: step 1310, determining the first ray and the second ray of the current shooting position; step 1320, determining the clarity of each candidate image; step 1330, judging whether the clarity is greater than a threshold; step 1340, acquiring a depth image again; step 1350, determining a key frame image; and step 1360, moving an image acquisition device.

[0086] In step 1310, taking the position of the image acquisition device in the world coordinate system as the starting point, the first ray is determined according to the shooting direction of the current shooting position; taking the position of the image acquisition device in the world coordinate system as the starting point, the ray pointing to the center point of the three-dimensional model in the plane where the target object is located is determined as the second ray.

[0087] The first ray represents the shooting direction information of the image acquisition device, and the second ray represents the current position information of the image acquisition device, which can be used as a basis for screening key frame images.

[0088] In some embodiments, the position and shooting direction of the image acquisition device in the world coordinate system are calculated according to external parameters of the image acquisition device at the current shooting position.

[0089] In some embodiments, the key frame image of the current shooting position is determined based on the first ray, the second ray, and the grid position corresponding to the current shooting position of the multiple depth images of the current shooting position. Figure 3 The embodiment in determines candidate images for key frame images.

[0090] Figure 3 Schematic diagrams showing some embodiments of the key frame image acquisition method disclosed in the present invention.

[0091] like Figure 3 As shown, the target object is completely covered in the three-dimensional hemispherical model, the first ray of the image acquisition device is l1, and the second ray is l2.

[0092] In some embodiments, the position p of the depth camera in the world coordinate system can be obtained according to the external parameters R and T. camera and the camera is facing d camera :

[0093]

[0094] Taking the current position of the depth camera as the endpoint, the direction of the depth camera is d camera As the direction, make ray l1 as the direction ray; take the current position of the depth camera as the endpoint, and the depth camera and the center of the sphere p o The connection p o -p camera The direction is ray l2 as the position ray. The key frame images can be filtered according to the grid corresponding to the current shooting position, the camera orientation, and the direction of the line connecting the camera and the center of the sphere.

[0095] In some embodiments, when the first ray and the second ray of a frame depth image at the current shooting position both pass through the grid corresponding to the current shooting position, the frame depth image is determined as the candidate image of the current shooting position. For example, the ray l1 and the ray l2 in the figure both pass through the grid corresponding to the current shooting position, indicating that the position ray and the orientation ray of the current depth camera are both aimed at the target object, and the corresponding depth image is determined as the candidate image.

[0096] After the candidate images are determined, Figure 2 In the remaining steps, a key frame image of the current shooting position is determined in each candidate image according to the clarity of each candidate image.

[0097] In step 1320, the definition of each candidate image is determined.

[0098] In some embodiments, edge detection operators (such as Laplace operator, Sobel operator, Canny operator, etc.) are used to calculate edge feature information of each candidate image; and the clarity of each candidate image is determined based on statistical characteristics of each edge feature information.

[0099] For example, the Laplace operator can be used to detect the clarity. The image can be converted into a grayscale image and processed using the Laplace operator; the variance of the Laplace operator processing result is calculated as the clarity of the current image.

[0100] In step 1330, it is determined whether the definition of each candidate image is greater than a threshold. If there is no candidate image with a definition greater than the threshold, step 1340 is executed; if there is a candidate image with a definition greater than the threshold, step 1350 is executed.

[0101] In step 1340 , multiple frames of depth images are acquired again at the current shooting position, and steps 1310 to 1330 are repeatedly performed until there is a candidate image with a clarity greater than a threshold.

[0102] In step 1350, candidate images with a definition greater than a threshold are determined as key frame images.

[0103] In step 1360, the image acquisition device is moved and multiple depth image frames are acquired again (step 1340). The next shooting position of the image acquisition device is taken as the current shooting position, and the corresponding key frame image is determined in the multiple depth image frames at the current shooting position (steps 1310 to 1360 are repeated).

[0104] In some embodiments, a 3D model rendered near the target object may serve as a UI guide for capturing key images.

[0105] In some embodiments, in response to the image acquisition device acquiring multiple frames of depth images at the current shooting position, a first marking process is performed on the corresponding grid of the current shooting position on the three-dimensional model. For example, when the depth camera is aimed at a certain shooting angle, the corresponding grid can be lit (such as changing color, etc.) to indicate the coverage angle of the currently acquired key frame image.

[0106] In some embodiments, in response to determining the key frame image of the current shooting position, a second marking process is performed on the corresponding grid of the current shooting position on the three-dimensional model to indicate that the key frame image of the current shooting position has been acquired.

[0107] For example, if the clarity is greater than the threshold, it will be saved as a key frame image, and the color of the corresponding grid of the key frame image will be changed, indicating that the key frame image has been collected at the current angle and it can be moved to the next shooting position to continue collecting. When key frame images are collected at all angles, all grids will change color, indicating that key frame images covering the full angle range have been obtained, and scanning can be completed as required.

[0108] Figure 4 A flowchart showing some embodiments of the method for 3D reconstruction of key frame images disclosed in the present invention.

[0109] like Figure 4 As shown, in step 410, the key frame images of the target object at each shooting position are collected using the key frame image collection method of any of the above embodiments.

[0110] In step 420, the target object is three-dimensionally reconstructed based on the key frame images of the target object at each shooting position.

[0111] In this way, since key frame images covering the complete angle range are obtained, the effect of three-dimensional reconstruction can be improved.

[0112] Figure 5 A block diagram showing some embodiments of a key frame image acquisition device of the present disclosure.

[0113] like Figure 5 As shown, the key frame image acquisition device 5 includes a point cloud determination unit 51 , a creation unit 52 , and a key frame determination unit 53 .

[0114] The point cloud determination unit 51 determines the point cloud data of the target object according to the multi-frame depth images of the target object acquired by the image acquisition device at the current shooting position and the internal parameters of the image acquisition device.

[0115] The building unit 52 builds a three-dimensional model that can surround the target object according to the position information of the point cloud data of the target object, the internal parameters of the image acquisition device and the external parameters at the current shooting position. The surface of the three-dimensional model is divided into multiple grids, each grid corresponding to a shooting position.

[0116] The key frame determining unit 53 determines the key frame image of the target object at the corresponding shooting position in the multi-frame depth image of each shooting position according to the grid corresponding to each shooting position and the shooting position and shooting direction of the multi-frame depth image acquired at each shooting position.

[0117] In some embodiments, the establishing unit 52 calculates the projection matrix of the three-dimensional model from the three-dimensional coordinate system to the two-dimensional coordinate system based on the internal parameters of the image acquisition device; and establishes the three-dimensional model based on the projection matrix, the position information of the vertices of each grid, and the external parameters of the image acquisition device at the shooting position corresponding to each grid.

[0118] In some embodiments, the building unit 52 performs plane fitting based on the point cloud data of the target object to determine the plane where the target object is located; and builds a three-dimensional model based on the position of the target object on the plane.

[0119] In some embodiments, the establishing unit 52 determines the position of the center of the circle according to the position of the target object on the plane model; and establishes a three-dimensional hemispherical model on the plane model as a three-dimensional model according to the position of the center of the circle.

[0120] In some embodiments, the key frame determination unit 53 determines the key frame image of the current shooting position according to the first ray, the second ray, and the grid position corresponding to the current shooting position of the multi-frame depth image of the current shooting position. The second ray is a ray starting from the position of the image acquisition device in the world coordinate system and pointing to the center point of the three-dimensional model in the plane where the target object is located, and the first ray points to the shooting direction of the image acquisition device at the current shooting position.

[0121] In some embodiments, when the first ray and the second ray of a frame depth image of the current shooting position both pass through the grid corresponding to the current shooting position, the key frame determination unit 53 determines the frame depth image as a candidate image of the current shooting position; and determines the key frame image of the current shooting position in each candidate image according to the clarity of each candidate image.

[0122] In some embodiments, the key frame determination unit 53 calculates edge feature information of each candidate image using an edge detection operator; and determines the clarity of each candidate image according to statistical characteristics of each edge feature information.

[0123] In some embodiments, when there are candidate images with clarity greater than a threshold, the key frame determination unit 53 determines the corresponding candidate images as key frame images, takes the next shooting position of the image acquisition device as the current shooting position, and determines the corresponding key frame images in the multi-frame depth images at the current shooting position; when there are no candidate images with clarity greater than the threshold, multiple-frame depth images are acquired again at the current shooting position until there is a candidate image with clarity greater than the threshold.

[0124] In some embodiments, the position and shooting direction of the image acquisition device in the world coordinate system are calculated according to external parameters of the image acquisition device at the current shooting position.

[0125] In some embodiments, the acquisition device 5 also includes: a marking unit 54, which is used to perform a first marking process on the corresponding grid of the current shooting position on the three-dimensional model in response to the image acquisition device acquiring multiple frames of depth images at the current shooting position; in response to determining the key frame image of the current shooting position, perform a second marking process on the corresponding grid of the current shooting position on the three-dimensional model to indicate that the key frame image of the current shooting position has been acquired.

[0126] In some embodiments, the acquisition device 5 further includes: a reconstruction unit 55, which is used to perform three-dimensional reconstruction of the target object according to the key frame images of the target object at each shooting position.

[0127] Figure 6 A block diagram showing some embodiments of a 3D reconstruction apparatus for key frame images of the present disclosure.

[0128] like Figure 6 As shown, the three-dimensional reconstruction device 6 of the key frame image includes: an acquisition unit 61, which is used to execute the key frame image acquisition method of any of the above-mentioned embodiments, and acquire the key frame images of the target object at each shooting position; a reconstruction unit 62, which is used to perform three-dimensional reconstruction of the target object based on the key frame images of the target object at each shooting position.

[0129] Figure 7 A block diagram showing some embodiments of the electronic device of the present disclosure.

[0130] like Figure 7 As shown, the electronic device 7 of this embodiment includes: a memory 71 and a processor 72 coupled to the memory 71, and the processor 72 is configured to execute the key frame image acquisition method or the three-dimensional reconstruction method of the key frame image in any of the above-mentioned embodiments based on the instructions stored in the memory 71.

[0131] The memory 71 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0132] Figure 8 A block diagram showing some other embodiments of the electronic device of the present disclosure.

[0133] like Figure 8 As shown, the electronic device 8 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810, and the processor 820 is configured to execute the key frame image acquisition method or the key frame image three-dimensional reconstruction method in any one of the aforementioned embodiments based on the instructions stored in the memory 810.

[0134] The memory 810 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0135] The electronic device 8 may also include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850, the memory 810, and the processor 820 may be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, a touch screen, a microphone, and a speaker. The network interface 840 provides a connection interface for various networked devices. The storage interface 850 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0136] Fig. 9 A block diagram showing some embodiments of a key frame image acquisition system of the present disclosure.

[0137] like Fig. 9 As shown, the key frame image acquisition system 9 includes: a key frame image acquisition device 91, used to execute the key frame image acquisition method in any of the above embodiments; an image acquisition device 92, used to acquire multiple frames of depth images of the target object at different shooting positions.

[0138] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media including but not limited to disk storage, CD-ROM, optical storage, etc., which contain computer-usable program code.

[0139] So far, the key frame image acquisition method, key frame image acquisition device, key frame image acquisition system, key frame image 3D reconstruction method, key frame image 3D reconstruction device and non-volatile computer readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0140] The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0141] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for collecting key frame images, comprising: Determine point cloud data of the target object according to multiple frames of depth images of the target object acquired by the image acquisition device at the current shooting position and internal parameters of the image acquisition device; Establishing a three-dimensional model capable of surrounding the target object according to the position information of the point cloud data of the target object, the internal parameters of the image acquisition device and the external parameters at the current shooting position, wherein the surface of the three-dimensional model is divided into a plurality of grids, each grid corresponding to a shooting position; According to the grids corresponding to the shooting positions, the shooting positions and shooting directions of the multi-frame depth images acquired at the shooting positions, a key frame image of the target object at the corresponding shooting position is determined in the multi-frame depth images at the shooting positions, Wherein, determining the key frame image of the target object at the corresponding shooting position from the multiple frames of depth images at the shooting positions includes: A key frame image of the current shooting position is determined based on a first ray, a second ray, and a grid position corresponding to the current shooting position of the multi-frame depth images of the current shooting position, wherein the first ray points to a shooting direction of the image acquisition device at the current shooting position, and the second ray takes the position of the image acquisition device in the world coordinate system as a starting point and points to a center point of the three-dimensional model in the plane where the target object is located, and both the first ray and the second ray of the key frame image are aimed at the target object.

2. The collection method according to claim 1, wherein: The step of establishing a three-dimensional model capable of surrounding the target object comprises: Calculating a projection matrix of the three-dimensional model from a three-dimensional coordinate system to a two-dimensional coordinate system according to internal parameters of the image acquisition device; The three-dimensional model is established according to the projection matrix, the position information of the vertices of each mesh, and the external parameters of the image acquisition device at the shooting position corresponding to each mesh.

3. The collection method according to claim 1, wherein: The step of establishing a three-dimensional model capable of surrounding the target object comprises: Performing plane fitting according to the point cloud data of the target object to determine the plane where the target object is located; The three-dimensional model is established according to the position of the target object on the plane.

4. The collection method according to claim 3, wherein: The establishing of the three-dimensional model according to the position of the target object on the plane comprises: Determining the position of the center of the circle according to the position of the target object on the plane; According to the position of the circle center, a three-dimensional hemispherical model is established on the plane as the three-dimensional model.

5. The collection method according to claim 1, wherein: The determining the key frame image of the current shooting position according to the first ray, the second ray of the multiple-frame depth images of the current shooting position, and the grid position corresponding to the current shooting position comprises: When the first ray and the second ray of a frame depth image of the current shooting position both pass through the grid corresponding to the current shooting position, determining the frame depth image as a candidate image of the current shooting position; According to the clarity of each candidate image, a key frame image of the current shooting position is determined in each candidate image.

6. The collection method according to claim 5, wherein: The step of determining the key frame image of the current shooting position from among the candidate images according to the clarity of the candidate images comprises: Calculating edge feature information of each candidate image using an edge detection operator; The clarity of each candidate image is determined according to the statistical characteristics of each edge feature information.

7. The collection method according to claim 5, wherein: Determining the key frame image of the target object at the corresponding shooting position from the multiple frames of depth images at the shooting positions comprises: In the case where there are candidate images with a definition greater than a threshold, the corresponding candidate images are determined as key frame images, the next shooting position of the image acquisition device is taken as the current shooting position, and the corresponding key frame images are determined in the multiple depth image frames at the current shooting position; When there is no candidate image with a definition greater than the threshold, multiple frames of depth images are acquired again at the current shooting position until there is a candidate image with a definition greater than the threshold.

8. The collection method according to claim 1, wherein: The position and shooting direction of the image acquisition device in the world coordinate system are calculated according to the external parameters of the image acquisition device at the current shooting position.

9. The collection method according to any one of claims 1 to 8, further comprising: In response to an image acquisition device acquiring a plurality of frames of depth images at the current shooting position, performing a first marking process on a corresponding grid of the current shooting position on the three-dimensional model; In response to determining the key frame image of the current shooting position, a second marking process is performed on the corresponding grid of the current shooting position on the three-dimensional model to indicate that the key frame image of the current shooting position has been collected.

10. The collection method according to any one of claims 1 to 8, further comprising: The target object is three-dimensionally reconstructed according to the key frame images of the target object at each shooting position.

11. A three-dimensional reconstruction method of a key frame image, comprising: Using the key frame image acquisition method described in any one of claims 1 to 9, the key frame images of the target object at each shooting position are acquired; The target object is three-dimensionally reconstructed according to the key frame images of the target object at each shooting position.

12. A key frame image acquisition device, comprising: A point cloud determination unit, configured to determine point cloud data of the target object based on a plurality of depth images of the target object acquired by an image acquisition device at a current shooting position and internal parameters of the image acquisition device; A building unit, configured to build a three-dimensional model capable of surrounding the target object according to the position information of the point cloud data of the target object, the internal parameters of the image acquisition device and the external parameters at the current shooting position, wherein the surface of the three-dimensional model is divided into a plurality of grids, each grid corresponding to a shooting position; a key frame determining unit, configured to determine a key frame image of the target object at a corresponding shooting position in the multi-frame depth image at each shooting position according to the grid corresponding to each shooting position, the shooting position and shooting direction of the multi-frame depth image acquired at each shooting position, Among them, the key frame determination unit determines the key frame image of the current shooting position based on the first ray, the second ray of the multi-frame depth image of the current shooting position, and the grid position corresponding to the current shooting position, the first ray points to the shooting direction of the image acquisition device at the current shooting position, the second ray is a ray starting from the position of the image acquisition device in the world coordinate system and pointing to the center point of the three-dimensional model in the plane where the target object is located, and the first ray and the second ray of the key frame image are both aimed at the target object.

13. The collection device according to claim 12, further comprising: A marking unit is used for performing a first marking process on a corresponding grid of the current shooting position on the three-dimensional model in response to an image acquisition device acquiring multiple frames of depth images at the current shooting position; and in response to determining a key frame image of the current shooting position, performing a second marking process on the corresponding grid of the current shooting position on the three-dimensional model to indicate that acquisition of the key frame image of the current shooting position is complete.

14. The collection device according to claim 12, further comprising: The reconstruction unit is used to perform three-dimensional reconstruction of the target object according to the key frame images of the target object at each shooting position.

15. A three-dimensional reconstruction device for a key frame image, comprising: A collection unit, used to execute the key frame image collection method according to any one of claims 1 to 9, and collect key frame images of the target object at each shooting position; The reconstruction unit is used to perform three-dimensional reconstruction of the target object according to the key frame images of the target object at each shooting position.

16. An electronic device, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the key frame image acquisition method described in any one of claims 1 to 10, or the three-dimensional reconstruction method of the key frame image described in claim 11, based on instructions stored in the memory.

17. A key frame image acquisition system, comprising: A key frame image acquisition device, used to execute the key frame image acquisition method described in any one of claims 1 to 10; The image acquisition device is used to acquire multiple frames of depth images of a target object at different shooting positions.

18. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the key frame image acquisition method described in any one of claims 1 to 10, or the three-dimensional reconstruction method of the key frame image described in claim 11.

Images