Three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The present disclosure relates to a three-dimensional reconstruction method, device, equipment and storage medium, the method comprising: obtaining an image sequence of an object to be reconstructed; determining the point cloud data of the image in the image sequence; generating an initial facet map corresponding to the historical frame image based on the point cloud data of the historical frame image corresponding to the current frame image in the image sequence; the current frame image is any frame image in the image sequence except the first frame image; generating a fused facet map corresponding to the current frame image based on the initial facet map and the point cloud data of the current frame image; generating a three-dimensional reconstruction result of the scene to be reconstructed based on the facet map corresponding to the first frame image and the fused facet map. The technical solution of the present disclosure can reduce the noise of the fusion result and improve the accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional reconstruction method, device, equipment and storage medium. Background Art

[0002] At present, the 3D reconstruction of scenes and objects has been widely used in many fields such as human-computer interaction, virtual reality, games and movies. In related technologies, when doing 3D reconstruction on a mobile terminal, the depth information of the image can be obtained, and the point cloud map can be determined based on the depth information and posture information, and the point cloud map can be fused to perform 3D reconstruction of the object in the image.

[0003] However, the flexibility of point cloud fusion in related technologies is poor, and the point cloud fusion process is prone to severe drift, resulting in large noise in the fusion result, thereby reducing the accuracy of 3D reconstruction. Summary of the invention

[0004] The present disclosure provides a 3D reconstruction method, device, equipment and storage medium to at least solve the problem that the flexibility of point cloud fusion in the related art is poor, and the point cloud fusion process is prone to serious drift, resulting in large noise in the fusion result, thereby reducing the accuracy of 3D reconstruction. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a three-dimensional reconstruction method is provided, comprising:

[0006] Acquire an image sequence of a scene to be reconstructed;

[0007] determining point cloud data of images in the image sequence;

[0008] Based on the point cloud data of the historical frame image corresponding to the current frame image in the image sequence, an initial facet map corresponding to the historical frame image is generated; the current frame image is any frame image except the first frame image in the image sequence, and the historical frame image is an image in the image sequence whose acquisition time point is before the current frame image;

[0009] Based on the initial surfel map and the point cloud data of the current frame image, generating a fused surfel map corresponding to the current frame image;

[0010] A three-dimensional reconstruction result of the scene to be reconstructed is generated according to the facet map corresponding to the first frame image and the fused facet map; the facet map corresponding to the first frame image is generated based on the point cloud data of the first frame image.

[0011] In an exemplary embodiment, generating a fused surfel map corresponding to the current frame image based on the initial surfel map and the point cloud data of the current frame image includes:

[0012] Projecting the initial surfel map onto the current frame image to obtain projected pixel points of the initial surfel map in the current frame image;

[0013] The fused surface element map is generated based on the projected pixel points and the point cloud data of the current frame image.

[0014] In an exemplary embodiment, the generating the fused surface element map based on the projected pixel points and the point cloud data of the current frame image includes:

[0015] Based on the point cloud data of the current frame image, three-dimensionally transform the coordinates of the projected pixel points and the coordinates of the pixel points in the current frame image to obtain a first new surface element map corresponding to the pixel points in the current frame image and a second new surface element map corresponding to the projected pixel points;

[0016] Fusing the initial surfel map and the second new surfel map to obtain a third new surfel map corresponding to the current frame image;

[0017] The first new surfel map and the third new surfel map are merged to obtain a fused surfel map corresponding to the current frame.

[0018] In an exemplary embodiment, the fusing the initial surfel map and the second new surfel map to obtain a third new surfel map corresponding to the current frame image includes:

[0019] Determining a positional relationship between the initial surfel map and the second new surfel map according to depth information of pixels in the historical frame image and depth information of pixels in the current frame image;

[0020] According to the positional relationship, the initial surfel map and the second new surfel map are fused to obtain the third new surfel map.

[0021] In an exemplary embodiment, the fusing the initial surfel map and the second new surfel map according to the positional relationship to obtain the third new surfel map includes:

[0022] When the position relationship indicates that the distance between the initial surfel map and the second new surfel map is less than or equal to a distance threshold, an average value of the initial surfel map and the second new surfel map is calculated to obtain the third new surfel map.

[0023] In an exemplary embodiment, the fusing the initial surfel map and the second new surfel map according to the positional relationship to obtain the third new surfel map includes:

[0024] In a case where the position relationship is used to indicate that the distance between the initial surfel map and the second new surfel map is greater than a distance threshold, the initial surfel map is replaced based on the second new surfel map to obtain the third new surfel map.

[0025] In an exemplary embodiment, the fusing the initial surfel map and the second new surfel map according to the positional relationship to obtain the third new surfel map includes:

[0026] When the positional relationship indicates that the initial surfel map is blocked by the second new surfel map, a new surfel map of the initial surfel map is created, and the created new surfel map and the second new surfel map are used as the third new surfel map.

[0027] In an exemplary embodiment, the method further comprises:

[0028] The facet map corresponding to the first frame image and the fused facet map are fused with the semantic segmentation result to obtain the semantically fused facet map corresponding to the image in the image sequence; the semantic segmentation result is obtained by semantically segmenting the map points, the map points are three-dimensional points that give depth information to the feature points in the key frame image, the key frame image is an image in the historical frame image that partially overlaps with the current frame image, and the feature points are obtained by feature extraction of the key frame image;

[0029] The semantically fused facet graph is subjected to gridding processing to obtain the three-dimensional reconstruction result.

[0030] In an exemplary embodiment, generating an initial surfel map corresponding to a historical frame image based on point cloud data of a historical frame image corresponding to a current frame image in the image sequence includes:

[0031] Based on the point cloud data of the historical frame image, coordinates of pixel points in the historical frame image are three-dimensionally transformed to obtain a facet map corresponding to the pixel points in the historical frame image;

[0032] The bin map corresponding to the pixel point is used as the initial bin map.

[0033] In an exemplary embodiment, determining the point cloud data of the images in the image sequence includes:

[0034] Determining depth information of pixel points of images in the image sequence and position information of images in the image sequence;

[0035] Point cloud data of the images in the image sequence are generated according to the depth information of the pixels of the image and the posture information.

[0036] In an exemplary embodiment, the posture information includes rotation posture information and translation posture information, and the method includes:

[0037] Obtaining world coordinate information of feature points in a previous frame image and rotational posture information of the current frame image; the previous frame image is an adjacent image in the image sequence whose acquisition time point is before the current frame image; the feature points in the previous frame image are obtained by extracting features from the previous frame image;

[0038] Performing optical flow tracking on the feature points in the previous frame image to determine image coordinate information of the feature points in the current frame image; the feature points in the current frame image are obtained by performing feature extraction on the current frame image;

[0039] Based on the world coordinate information, the image coordinate information and the rotational pose information of the current frame image, the translational pose information of the current frame image is determined.

[0040] In an exemplary embodiment, the method further includes a step of adjusting the translation posture information, and the step of adjusting the translation posture information includes:

[0041] Acquire a key frame image from the historical frame image; the key frame image partially overlaps with the current frame image;

[0042] Constructing a local map using feature points in the key frame image; the feature points in the key frame image are obtained by extracting features from the key frame image;

[0043] Matching pixels in the current frame image based on the local map;

[0044] The translation pose information is adjusted based on the matching result to obtain the adjusted translation pose information of the current frame image.

[0045] In an exemplary embodiment, generating point cloud data of images in the image sequence according to the depth information of the pixels of the image and the pose information includes:

[0046] Acquire first position information of the image collector corresponding to the first frame image;

[0047] Determine the second position information of the image collector corresponding to any frame of image according to the depth information, the posture information and the first position information of the pixel points in any one of the images;

[0048] Determine world coordinate information of a pixel point in the image according to the position information corresponding to the image and the depth information of the pixel point in the image; the position information corresponding to the image includes first position information and second position information;

[0049] Generating point cloud data of the image according to world coordinate information of pixel points in the image;

[0050] Point cloud data of the image is generated according to world coordinate information of pixel points in the image.

[0051] According to a second aspect of an embodiment of the present disclosure, there is provided a three-dimensional reconstruction device, comprising:

[0052] An image sequence acquisition module is configured to acquire an image sequence of a scene to be reconstructed;

[0053] A point cloud data generation module, configured to determine point cloud data of an image in the image sequence;

[0054] The initial facet map generation module is configured to generate an initial facet map corresponding to a historical frame image based on point cloud data of a historical frame image corresponding to a current frame image in the image sequence; the current frame image is any frame image except the first frame image in the image sequence, and the historical frame image is an image in the image sequence whose acquisition time point is before the current frame image;

[0055] A fused surfel map generating module is configured to generate a fused surfel map corresponding to the current frame image based on the initial surfel map and the point cloud data of the current frame image;

[0056] The three-dimensional reconstruction result generation module is configured to generate a three-dimensional reconstruction result of the scene to be reconstructed based on the facet map corresponding to the first frame image and the fused facet map; the facet map corresponding to the first frame image is generated based on the point cloud data of the first frame image.

[0057] In an exemplary embodiment, the fused surface element graph generation module includes:

[0058] A projection submodule is configured to project the initial surfel image into the current frame image to obtain projection pixel points of the initial surfel image in the current frame image;

[0059] The generating submodule is configured to generate the fused surface element map based on the projected pixel points and the point cloud data of the current frame image.

[0060] In an exemplary embodiment, the generating submodule includes:

[0061] a conversion unit configured to perform a three-dimensional conversion on the coordinates of the projected pixel points and the coordinates of the pixel points in the current frame image based on the point cloud data of the current frame image, so as to obtain a first new surface element map corresponding to the pixel points in the current frame image and a second new surface element map corresponding to the projected pixel points;

[0062] A third new surfel map determining unit is configured to fuse the initial surfel map and the second new surfel map to obtain a third new surfel map corresponding to the current frame image;

[0063] The fused surfel map determining unit is configured to merge the first new surfel map and the third new surfel map to obtain a fused surfel map corresponding to the current frame.

[0064] In an exemplary embodiment, the third new surfel map determining unit includes:

[0065] a position relationship determination subunit, configured to determine the position relationship between the initial surfel map and the second new surfel map according to the depth information of the pixel points in the historical frame image and the depth information of the pixel points in the current frame image;

[0066] The fusion relationship determination subunit is configured to perform fusion of the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map.

[0067] In an exemplary embodiment, the fusion subunit is configured to calculate the average value of the initial surfel map and the second new surfel map to obtain the third new surfel map when the position relationship indicates that the distance between the initial surfel map and the second new surfel map is less than or equal to a distance threshold.

[0068] In an exemplary embodiment, the fusion subunit is configured to replace the initial surfel map based on the second new surfel map to obtain the third new surfel map when the position relationship indicates that the distance between the initial surfel map and the second new surfel map is greater than a distance threshold.

[0069] In an exemplary embodiment, the fusion subunit is configured to create a new facet map of the initial facet map when the positional relationship indicates that the initial facet map is occluded by the second new facet map, and use the created new facet map and the second new facet map as the third new facet map.

[0070] In an exemplary embodiment, the device further comprises:

[0071] A semantic fusion module is configured to fuse the facet map corresponding to the first frame image and the fused facet map with the semantic segmentation result to obtain a semantically fused facet map corresponding to the image in the image sequence; the semantic segmentation result is obtained by semantically segmenting a map point, the map point is a three-dimensional point that gives depth information to a feature point in a key frame image, the key frame image is an image in the historical frame image that partially overlaps with the current frame image, and the feature point is obtained by feature extraction of the key frame image;

[0072] The gridding processing module is configured to perform gridding processing on the semantically fused facet graph to obtain the three-dimensional reconstruction result.

[0073] In an exemplary embodiment, the initial surface element graph generation module includes:

[0074] A pixel bin map generating unit is configured to perform a three-dimensional transformation on the coordinates of the pixel points in the historical frame image based on the point cloud data of the historical frame image, so as to obtain a bin map corresponding to the pixel points in the historical frame image;

[0075] The initial surfel map determining unit is configured to execute the surfel map corresponding to the pixel point as the initial surfel map.

[0076] In an exemplary embodiment, the point cloud data generation module includes:

[0077] A depth and pose determination unit, configured to determine depth information of pixels of images in the image sequence and pose information of images in the image sequence;

[0078] The point cloud data generating unit is configured to generate point cloud data of the images in the image sequence according to the depth information of the pixel points of the image and the posture information.

[0079] In an exemplary embodiment, the posture information includes rotation posture information and translation posture information, and the device includes:

[0080] A rotation posture information acquisition module is configured to acquire world coordinate information of feature points in a previous frame image and rotation posture information of the current frame image; the previous frame image is an adjacent image in the image sequence whose acquisition time point is before the current frame image; the feature points in the previous frame image are obtained by extracting features from the previous frame image;

[0081] An optical flow tracking module is configured to perform optical flow tracking on the feature points in the previous frame image to determine image coordinate information of the feature points in the current frame image; the feature points in the current frame image are obtained by extracting features from the current frame image;

[0082] The translation posture information determination module is configured to determine the translation posture information of the current frame image based on the world coordinate information, the image coordinate information and the rotation posture information of the current frame image.

[0083] In an exemplary embodiment, the device further includes an adjustment module for adjusting the translation posture information, and the adjustment module includes:

[0084] A key frame image acquisition unit is configured to acquire a key frame image from the historical frame image; the key frame image partially overlaps with the current frame image;

[0085] A local map construction unit is configured to construct a local map using feature points in the key frame image; the feature points in the key frame image are obtained by extracting features from the key frame image;

[0086] A matching unit, configured to perform matching of pixel points in the current frame image based on the local map;

[0087] The posture adjustment unit is configured to adjust the translation posture information based on the matching result to obtain the adjusted translation posture information of the current frame image.

[0088] In an exemplary embodiment, the point cloud data generating unit includes:

[0089] A first position information acquisition subunit is configured to acquire first position information of the image collector corresponding to the first frame image;

[0090] A second position information determining subunit is configured to determine the second position information of the image collector corresponding to any frame of image according to the depth information, the position information and the first position information of the pixel points in any image;

[0091] a world coordinate information determination subunit, configured to determine the world coordinate information of the pixel point in the image according to the position information corresponding to the image and the depth information of the pixel point in the image; the position information corresponding to the image includes the first position information and the second position information;

[0092] The point cloud data generating subunit is configured to generate the point cloud data of the image according to the world coordinate information of the pixel points in the image.

[0093] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device for three-dimensional reconstruction, including:

[0094] processor;

[0095] A memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the three-dimensional reconstruction method as described above.

[0096] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the three-dimensional reconstruction method as described above.

[0097] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the three-dimensional reconstruction method described above is implemented.

[0098] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0099] The disclosed embodiment acquires an image sequence of a scene to be reconstructed; determines point cloud data of images in the image sequence; generates an initial facet map corresponding to the historical frame image based on point cloud data of the historical frame image corresponding to the current frame image in the image sequence; the current frame image is any frame image except the first frame image in the image sequence, and a fused facet map corresponding to the current frame image is generated based on the initial facet map and the point cloud data of the current frame image; a three-dimensional reconstruction result of the scene to be reconstructed is generated based on the facet map corresponding to the first frame image and the fused facet map, thereby effectively avoiding drift in the process of point cloud map fusion, reducing noise in the fusion result, and improving the accuracy of three-dimensional reconstruction.

[0100] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0102] Figure 1 It is a schematic diagram of an implementation environment of a three-dimensional reconstruction method according to an exemplary embodiment.

[0103] Figure 2 The figure is a flow chart of a 3D reconstruction method according to an exemplary embodiment.

[0104] Figure 3 is a schematic diagram of a deep network model according to an exemplary embodiment.

[0105] Figure 4 The present invention is a flowchart of generating an initial facet according to an exemplary embodiment.

[0106] Figure 5 The present invention is a flowchart showing a method of generating a fused surface element map corresponding to a current frame image according to an exemplary embodiment.

[0107] Figure 6 The present invention is a flowchart of generating a fused facet graph according to an exemplary embodiment.

[0108] Figure 7 The present invention is a flowchart showing a method of obtaining a third new bin map corresponding to a current frame image according to an exemplary embodiment.

[0109] Figure 8 FIG. 1 is a schematic diagram of facet fusion according to an exemplary embodiment. Figure 1 .

[0110] Fig. 9 FIG. 1 is a schematic diagram of facet fusion according to an exemplary embodiment. Figure 2 .

[0111] Fig.10 The figure is a flowchart of semantic fusion according to an exemplary embodiment.

[0112] Fig.11 The figure is a block diagram of a 3D reconstruction device according to an exemplary embodiment.

[0113] Fig.12 It is a block diagram of an electronic device for three-dimensional reconstruction according to an exemplary embodiment. DETAILED DESCRIPTION

[0114] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0115] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0116] Figure 1 FIG. 1 is a schematic diagram showing an implementation environment of a three-dimensional reconstruction method according to an exemplary embodiment. Figure 1 As shown, the implementation environment may include at least a terminal 01 and a server 02. The terminal 01 and the server 2 may be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitation thereto.

[0117] Specifically, the terminal can be used to collect an image sequence of a scene to be reconstructed. Optionally, the terminal 01 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart watch, etc., but is not limited thereto.

[0118] Specifically, the server 02 can be used to obtain an image sequence of a scene to be reconstructed; determine the point cloud data of an image in the image sequence; generate an initial facet map corresponding to the historical frame image based on the point cloud data of the historical frame image corresponding to the current frame image in the image sequence; generate a fused facet map corresponding to the current frame image based on the initial facet map and the point cloud data of the current frame image; generate a three-dimensional reconstruction result of the scene to be reconstructed based on the facet map and the fused facet map corresponding to the first frame image. Optionally, the server 02 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0119] It should be noted that Figure 1 This is just an example. In other scenarios, other implementation environments may also be included. For example, the implementation environment may include a terminal, through which an image sequence of a scene to be reconstructed is sampled, and a three-dimensional reconstruction result of the scene to be reconstructed is generated.

[0120] Figure 2 is a flow chart of a three-dimensional reconstruction method according to an exemplary embodiment. Figure 2 As shown, the following steps are included.

[0121] In step S11, an image sequence of a scene to be reconstructed is acquired.

[0122] The embodiments of the present disclosure may acquire an image sequence of a scene to be reconstructed through various image acquisition devices, which are not specifically limited herein.

[0123] In an exemplary embodiment, the image sequence may be a continuous image frame obtained by a single-purpose image collector from various angles to collect images of an object in a scene to be reconstructed. The single-purpose image collector may be, for example, a single camera on a mobile computing device. By collecting image sequences with a single-purpose image collector, the defect of limited range of a depth sensor can be overcome, the cost can be reduced, and the adaptability and scalability are good.

[0124] Exemplarily, the object in the scene to be reconstructed may be, for example, any object or any spatial region.

[0125] In step S13, point cloud data of the images in the above image sequence are determined.

[0126] In an exemplary embodiment, determining point cloud data of an image in the image sequence may include:

[0127] Determine the depth information of the pixels of the images in the above image sequence and the position and posture information of the images in the above image sequence.

[0128] Point cloud data of the images in the image sequence are generated based on the depth information of the pixels of the image and the posture information.

[0129] Generating point cloud data through the depth information and posture information of the image pixels can improve the efficiency and accuracy of point cloud data generation.

[0130] Figure 3 is a schematic diagram of a deep network model according to an exemplary embodiment. Figure 3 As shown, in an optional embodiment, the images in the image sequence can be input into a preset deep network model to obtain the depth information of the pixel points of the image. Figure 3 In the structure of the deep network model, the feature extraction module, feature fusion module and prediction module can be included. The feature extraction module is mainly used to extract features from the bottom layer to the top layer of the image, and to provide features for the feature fusion module; the feature fusion module is used to gradually restore the image resolution and reduce the number of channels, and fuse the high-level features and bottom-level features extracted by the feature extraction module to obtain fused features; the prediction module is mainly used to predict the depth value of the pixel in the image based on the features of each pixel in the fused features. By extracting the depth information of the pixel points of the image through the above three modules, the extraction efficiency and accuracy of the depth information can be improved.

[0131] Exemplarily, the training method of the deep network model can be: obtaining training data, which may include: each sample image and corresponding depth information; using each sample image and corresponding depth information to train the initial deep network model to obtain a trained deep network model.

[0132] The depth information of the above-mentioned image refers to: the distance offset information of each pixel point in the image relative to the image collector on a fixed coordinate axis or a fixed orientation determined by multiple coordinate axes.

[0133] In an optional embodiment, the above-mentioned posture information includes rotation posture information and translation posture information, and obtaining the posture information of the image in the image sequence may include:

[0134] Obtain the world coordinate information of the feature points in the previous frame image and the rotation posture information of the current frame image; the previous frame image is an adjacent image in the image sequence, whose acquisition time point is before the current frame image; the feature points in the previous frame image are obtained by feature extraction of the previous frame image.

[0135] The optical flow tracking is performed on the feature points in the above-mentioned previous frame image to determine the image coordinate information of the feature points in the above-mentioned current frame image; the feature points in the above-mentioned current frame image are obtained by performing feature extraction on the above-mentioned current frame image.

[0136] Based on the world coordinate information, the image coordinate information and the rotational posture information of the current frame image, the translational posture information of the current frame image is determined.

[0137] In an exemplary embodiment, an inertial measurement unit (IMU) may be configured on a single-purpose image collector to measure the inertial measurement information of the image collector in real time, so as to obtain the inertial measurement information of the image collector when collecting the image to be processed. Among them, the inertial measurement information may include rotational posture information. It should be noted that the above-mentioned rotational posture information may be the angular offset information of the first posture of the image collector relative to the second posture. Among them, the first posture may be the posture of the image collector when capturing an image; the second posture may be the posture of the image collector when capturing the first frame of the image.

[0138] In the above embodiment, by setting an inertial measurement unit on a single-purpose image collector, the rotational posture information when the image collector collects images can be directly obtained, and then the translation posture information can be conveniently determined by combining the rotational posture information, thereby improving the rate and accuracy of determining the translation posture information. In an exemplary embodiment, since the position of the object in the scene to be reconstructed is fixed and does not move with the movement of the image collector, the world coordinate information of the object in the scene to be reconstructed in multiple images continuously captured by the image collector is the same. In addition, during the continuous shooting process of the image collector, the posture change of the image collector when shooting two adjacent images is limited, and usually there will be no significant changes. Therefore, a six-degree-of-freedom posture constraint can be constructed based on this, and the translation posture information can be solved based on the six-degree-of-freedom posture constraint, which can improve the accuracy of solving the translation posture information. Among them, the six degrees of freedom refer to the freedom of movement along the three rectangular coordinate axes of x, y, and z in the world coordinate system and the freedom of rotation around these three rectangular coordinate axes. Therefore, the world coordinate information of each feature point in the previous frame image can be obtained; the optical flow tracking is performed on each feature point in the current frame image to determine the image coordinate information of each feature point in the current frame image; the translation pose information of the current frame image is used as a variable, the world coordinate information of each feature point in the previous frame image, the image coordinate information of each feature point in the current frame image, and the rotation pose information of the current frame image are used as parameters, and the six-degree-of-freedom pose constraint is used as a condition to construct a set of equations to solve the translation pose information of the current frame image. Among them, the world coordinate information refers to the absolute coordinate system of the system, which defines the objective space of the physical world and is used as a reference coordinate system to describe the position of the image collector and the object; the horizontal rightward of the image coordinate information is the x-axis, the vertical downward is the y-axis, and the origin is at the upper left.

[0139] In the above embodiment, the algorithm for solving the translation pose information of the image using the above-mentioned solution principle may be, for example, a multi-point perspective imaging algorithm (Pespective-N-Point, PNP). Among them, the method for extracting each feature point in the image may be: extracting feature points of the image using an intersection detection algorithm (Good Feature To Track, GFTT) to obtain each feature point in the image. It should be noted that the translation pose information is the position offset information of the first pose of the image collector relative to the second pose. Among them, the first pose is the pose of the image collector when it collects the image to be processed; the second pose is the pose of the image collector when it collects the first frame image in the image sequence.

[0140] In a feasible embodiment, the method further includes a step of adjusting the translation posture information, and the step of adjusting the translation posture information may include:

[0141] A key frame image is obtained from the above historical frame image; the above key frame image partially overlaps with the above current frame image.

[0142] The local map is constructed using the feature points in the key frame image; the feature points in the key frame image are obtained by extracting features from the key frame image.

[0143] The pixel points in the current frame image are matched based on the local map.

[0144] The above-mentioned translation posture information is adjusted based on the matching result to obtain the adjusted translation posture information of the above-mentioned current frame image.

[0145] In the above embodiment, a current frame image whose field of view partially overlaps with the field of view of the current frame image can be extracted from the historical frame image. The field of view can refer to the field of view that the image can see. The key frame is subjected to GFTT feature point extraction to obtain each feature point in the key frame image. The local map is constructed using each feature point in the key frame image. The map point in the local map refers to the three-dimensional point after each feature point in the key frame image is given depth, wherein each feature point in the key frame image is obtained by feature extraction of the key frame image. The map point in the local map is matched with the pixel point of the current image frame, for example, the matching of the map point and the pixel point is completed by using the optical flow method. The matching premise is to find the matching relationship between the map point and the pixel point of the current frame, and use the matching relationship to optimize the translation posture by reducing the reprojection error of the map point. Wherein, the reprojection error refers to: the geometric pixel distance between the projection point p1 of the map point P in the current frame image and the pixel point p2 matching P with the current frame image.

[0146] In the disclosed embodiment, by adjusting the translation posture of the current frame image through the local map constructed by the key frame image, the accuracy of determining the translation posture information can be improved, thereby improving the accuracy of determining the three-dimensional reconstruction result.

[0147] In an optional embodiment, generating point cloud data of the images in the image sequence according to the depth information and the posture information may include:

[0148] The first position information of the image collector corresponding to the first frame image is obtained.

[0149] The second position information of the image collector corresponding to any of the frame images is determined according to the depth information, the posture information and the first position information of the pixel points in any of the above images.

[0150] The world coordinate information of the pixel points in the image is determined according to the position information corresponding to the image and the depth information of the pixel points in the image; the position information corresponding to the image includes the first position information and the second position information.

[0151] The point cloud data of the image is generated according to the world coordinate information of the pixel points in the image.

[0152] In the above embodiment, the rotational posture information and translational posture information of the image are the offset information between the first posture and the second posture, wherein the first posture is the posture of the image collector when collecting any image; the second posture is the posture of the image collector when collecting the first frame image. Therefore, in combination with the rotational posture information, translational posture information of the image and the position information of the image collector corresponding to the first frame image, the position information of the image collector corresponding to any image can be determined. In the above embodiment, the depth information of the pixel points in the image refers to the distance offset information of each pixel point in the image relative to the image collector on a fixed coordinate axis or a fixed orientation determined by multiple coordinate axes. Therefore, in combination with the position information of the image collector corresponding to the image and the depth information of the pixel points in the image, the world coordinate information of each pixel point in the image can be determined.

[0153] In the embodiment of the present disclosure, based on the depth information, rotational posture information, and translational posture information of the pixel points in each frame image in the image sequence, the world coordinate information of each pixel point in each frame image can be directly determined, and then the world coordinate information of each pixel point in each frame image can be combined to generate point cloud data, which can improve the rate and accuracy of generating point cloud data.

[0154] In step S15, based on the point cloud data of the historical frame image corresponding to the current frame image in the above-mentioned image sequence, an initial facet map corresponding to the above-mentioned historical frame image is generated; the above-mentioned current frame image is any frame image in the above-mentioned image sequence except the first frame image, and the above-mentioned historical frame image is an image in the image sequence whose acquisition time point is before the above-mentioned current frame image.

[0155] Specifically, any frame image except the first frame image in the image sequence can be used as the current frame image, and the historical frame image is an image in the image sequence whose acquisition time point is before the current frame image.

[0156] The present disclosure may generate an initial facet map in a variety of ways, which are not specifically limited herein. Figure 4 is a flow chart showing a method of generating an initial face element according to an exemplary embodiment. Figure 4 As shown, in an optional embodiment, in the above step S15, generating the initial surfel map corresponding to the above historical frame image based on the point cloud data of the historical frame image corresponding to the current frame image in the above image sequence may include:

[0157] In step S1501, based on the point cloud data of the historical frame image, the coordinates of the pixel points in the historical frame image are three-dimensionally transformed to obtain a surface element map corresponding to the pixel points in the historical frame image.

[0158] In step S1503, the bin map corresponding to the pixel point is used as the initial bin map.

[0159] Among them, the facet map refers to: replacing the original object surface with a plane or a curved surface. The surface to be reconstructed is composed of multiple facets. Each facet map stores the three-dimensional coordinate point P, color information c (R, G, B), position information (X, Y, Z), facet radius r, normal vector n, weight w, confidence σ, creation timestamp of the three-dimensional coordinate point, and the last update timestamp.

[0160] In one way, in the above step S1501, the point cloud data of the historical image frame (including pose information, depth map information, color information, etc.) can be used to perform a three-dimensional transformation on the coordinates of each pixel in the historical image frame to obtain a facet map corresponding to each pixel in the historical frame image, that is, to obtain a facet map equal to the number of pixels. The facet map corresponding to each pixel is the initial facet map of the historical image frame. Wherein, the confidence of each facet is set to 1, and the calculation formula of the radius rs is as follows:

[0161]

[0162] Among them, p(x,y) is the pixel point corresponding to each face element, and u and v refer to the coordinates of the pixel point.

[0163] Exemplarily, the coordinates of the pixel points can be transformed into three dimensions in the following manner: Assuming that the pose information is T, the coordinates of the pixel points are p(u, v, 1), and the pixel depth is d, the conversion formula for a single pixel point can be:

[0164] T*(d*K^(-1)*p),

[0165] Where K is the intrinsic parameter matrix [fx,0,cx,0,fy,cy,0,0,1].

[0166] The following is an example to illustrate the above steps S1501 to S1503:

[0167] Assuming that the current frame image is the second frame image in the image sequence, the historical frame image is the first frame image, and the first frame image contains 10 pixels. The point cloud data of the first frame image (including posture information, depth map information, color information, etc.) can be used to perform three-dimensional transformation on the coordinates of these 10 pixels to obtain the surface element maps corresponding to each of these 10 pixels, that is, to obtain the surface element maps equal to the number of these 10 pixels, and use the surface element maps corresponding to each of these 10 pixels as the initial surface element map of the first frame image.

[0168] Assuming that the current frame image is the third frame image in the image sequence, the historical frame images include the first frame image and the second frame image, the first frame image contains 10 pixels, and the second frame image contains 15 pixels. Then, the point cloud data of the first frame image (including posture information, depth map information, color information, etc.) can be used to perform three-dimensional transformation on the coordinates of the 10 pixels in the first frame, and obtain the surface element maps corresponding to each of the 10 pixels, that is, obtain the surface element maps equal to the number of these 10 pixels, and use the surface element maps corresponding to each of the 10 pixels as the initial surface element map of the first frame image. Using the point cloud data of the second frame image (including posture information, depth map information, color information, etc.), the coordinates of the 15 pixels in the second frame image are respectively transformed in three dimensions, and the surface element maps corresponding to each of the 15 pixels are obtained, that is, obtain the surface element maps equal to the number of these 15 pixels, and use the surface element maps corresponding to each of the 15 pixels as the initial surface element map of the second frame image.

[0169] In the disclosed embodiment, the coordinates of the pixel points in the historical frame image are converted into three dimensions through the point cloud data of the historical frame image to obtain a facet map corresponding to the pixel points in the historical frame image, which can improve the accuracy and efficiency of facet map creation, and further improve the accuracy and efficiency of three-dimensional reconstruction.

[0170] In step S17, a fused surfel map corresponding to the current frame image is generated based on the initial surfel map and the point cloud data of the current frame image.

[0171] Figure 5 FIG. 1 is a flow chart showing a method of generating a fused facet map corresponding to a current frame image according to an exemplary embodiment. Figure 5 As shown, in an optional embodiment, in the above step S17, generating a fused surfel map corresponding to the current frame image based on the above initial surfel map and the point cloud data of the current frame image may include:

[0172] In step S1701, the initial surfel map is projected into the current frame image to obtain the projected pixel points of the initial surfel map in the current frame image.

[0173] Exemplarily, if the current frame image is the second frame image in the image sequence, the initial facet map corresponding to each pixel point in the first frame image (i.e., the old facet map of the first frame image) can be projected to the second frame image to obtain the initial facet map corresponding to each pixel point in the first frame image and the projected pixel point in the second frame.

[0174] If the current frame image is the third frame image, the initial facet map corresponding to each pixel point in the first frame image (i.e., the old facet map of the first frame image) and the initial facet map corresponding to each pixel point in the second frame image (i.e., the old facet map of the second frame image) can be projected into the third frame image to obtain the projected pixel points of the old facet map of the first frame image in the third frame image and the projected pixel points of the old facet map of the second frame image in the third frame image.

[0175] It should be noted that, when the current frame image is the third frame image, in one way, the old surfel map of the second frame image can be obtained by three-dimensionally transforming the coordinates of the pixel points in the second frame image through the point cloud data of the second frame image. In another way, when the current frame image is the third frame image, since the second frame image has been subjected to surfel fusion, the corresponding fused surfel map of the second frame image can be used as the old surfel map of the second frame.

[0176] In step S1703, a fused surfel map is generated based on the projected pixel points and the point cloud data of the current frame image.

[0177] In the disclosed embodiment, a fused facet map is generated by projecting pixels of the initial facet and point cloud data of the current image frame, so that the same pixel can be fused to generate a dense surface, reduce fusion noise, and improve the generation accuracy of the fused facet map, thereby improving the accuracy of 3D reconstruction. At the same time, the initial facet of the historical frame image is projected into the current frame image, and the fused facet map is generated by projecting pixels and point cloud data of the current image frame, so that the flexibility of point cloud data fusion can be improved. Furthermore, the resolution of the 3D reconstruction result is related to the pixels. The more pixels there are, the higher the resolution and the clearer the 3D reconstruction result. Since the above fusion process is related to the projected pixels, and the projected pixels are related to the historical frame image, the number of which is not fixed but adjustable. The fused facet map is generated by adjustable projected pixels and point cloud data, so that the dense surface output by the 3D reconstruction has the characteristic of adjustable resolution. In addition, by generating the fused facet map in the above manner, a large amount of global surface information can be output on the mobile terminal.

[0178] Figure 6 FIG. 1 is a flow chart showing a method of generating a fused facet graph according to an exemplary embodiment. Figure 6As shown, in an optional embodiment, in the above step S1703, generating a fused surface element map based on the above projected pixel points and the point cloud data of the above current frame image may include:

[0179] In step S17031, based on the point cloud data of the current frame image, the coordinates of the projected pixel points and the coordinates of the pixel points in the current frame image are three-dimensionally transformed to obtain a first new facet map corresponding to the pixel points in the current frame image and a second new facet map corresponding to the projected pixel points.

[0180] In step S17033, the initial surfel map and the second new surfel map are fused to obtain a third new surfel map corresponding to the current frame image.

[0181] In step S17035, the first new surfel map and the third new surfel map are merged to obtain a fused surfel map corresponding to the current frame.

[0182] In the above embodiment, since the historical frame image is projected onto the current frame image to obtain projected pixel points, the projected pixel points are three-dimensionally transformed through the point cloud data of the current frame image, and a second new facet map corresponding to the projected pixel points can be obtained. The current frame image itself still has pixel points, and the various pixel points in the current frame image itself are three-dimensionally transformed through the point cloud data of the current frame image, and a first new facet map corresponding to the various pixel points in the current frame image itself can be obtained. For the projected pixel point, there is both an old facet map (i.e., the initial facet map corresponding to the projected pixel point) and a new facet map (i.e., the second new facet map), that is, there are two facet maps for one pixel point. The old facet map and the new facet map of the projected pixel point can be fused to obtain a third new facet map, and the first new facet map and the third new facet map are used as the fused facet map corresponding to the current frame, thereby realizing the fusion of the same pixel point to generate a dense surface, improving the determination accuracy of the facet of the same pixel point, reducing the fusion noise, and thus improving the accuracy of three-dimensional reconstruction; in addition, the resolution of the three-dimensional reconstruction result is related to the pixel points, the more pixels there are, the higher the resolution, and the clearer the three-dimensional reconstruction result. Since the above fusion process is related to the projected pixel points, and the projected pixel points are related to the historical frame image, the number of which is not fixed but adjustable. By fusing the adjustable second new facet map of the projected pixel point with the initial facet map, the dense surface output by the three-dimensional reconstruction has the characteristic of adjustable resolution.

[0183] Figure 7 FIG. 1 is a flow chart showing a method of obtaining a third new bin map corresponding to a current frame image according to an exemplary embodiment. Figure 7As shown, in an optional embodiment, in the step S17033, the fusion of the initial surfel map and the second new surfel map to obtain the third new surfel map corresponding to the current frame image may include:

[0184] In step S170331, the positional relationship between the initial surfel map and the second new surfel map is determined based on the depth information of the pixels in the historical frame image and the depth information of the pixels in the current frame image.

[0185] In step S170333, the initial surfel map and the second new surfel map are fused according to the positional relationship to obtain a third new surfel map.

[0186] In this embodiment, since the projection pixel point has both an old facet map (i.e., the initial facet map corresponding to the projection pixel point) and a new facet map (i.e., the second new facet map), that is, there are two facet maps for one pixel point, the fusion relationship between the two facet maps can be determined by the relative position between the two facet maps, and the new facet map and the old facet map of the same pixel point can be fused based on the fusion relationship.

[0187] In an exemplary embodiment, since the depth information of the pixel points in the image refers to: the distance offset information of each pixel point in the image relative to the image collector on a fixed coordinate axis or a fixed orientation determined by multiple coordinate axes. Therefore, according to the depth information of the pixel points in the historical frame image and the depth information of the pixel points in the current frame image, the relative position relationship between the second new facet map and the old facet map can be determined, and the fusion relationship between the two facet maps can be determined according to the relative position relationship. For example, if the distance between the second new facet map and the old facet map is less than or equal to the distance threshold according to the relative position between the two, the fusion relationship can be that the second new facet map is compatible with the old facet map (that is, the old facet map is supported by the new facet map); if the distance between the second new facet map and the old facet map is greater than the distance threshold according to the relative position between the two, the fusion relationship can be that the second new facet map conflicts with the old facet map; if the old facet map is blocked by the second new facet map according to the relative position between the second new facet map and the old facet map, the fusion relationship can be that the old facet map is blocked by the second new facet map.

[0188] Figure 8 FIG. 1 is a schematic diagram of facet fusion according to an exemplary embodiment. Figure 1 .like Figure 8As shown in the figure, number "1" represents the new surfel map, number "2" represents the surfel map that is not considered outside the camera's field of view, number "3" represents the support surfel map that is compatible with the second new surfel map, number "4" represents the conflict surfel map that conflicts with the second new surfel, and number "5" represents the occlusion surfel that is blocked by the second new surfel. Among them, the conflict surfel map, support surfel map, occlusion surfel map, and surfel map that is not considered are all old surfel maps.

[0189] In the disclosed embodiment, for the new facet map and the old facet map of the same pixel point, the fusion relationship can be determined according to the positional relationship between the two, and the new facet map and the old facet map of the same pixel point are fused according to the fusion relationship, thereby realizing the fusion of the same pixel point to generate a dense surface, improving the determination accuracy of the facet map of the same pixel point, reducing fusion noise, and thus improving the accuracy of three-dimensional reconstruction; in addition, according to the positional relationship between the two, a rich fusion relationship (for example, support, conflict, occlusion, etc.) can be determined, and the new facet map and the old facet map are fused through the rich fusion relationship, thereby improving the flexibility of facet fusion.

[0190] In an exemplary embodiment, in the above step S170333, the above merging the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map may include:

[0191] When the positional relationship indicates that the distance between the initial surfel map and the second new surfel map is less than or equal to a distance threshold, an average value of the initial surfel map and the second new surfel map is calculated to obtain the third new surfel map.

[0192] Fig. 9 FIG. 1 is a schematic diagram of facet fusion according to an exemplary embodiment. Figure 2 .like Fig. 9 As shown, for the support facet graph ( Figure 8 The code "3" in the figure can be used to calculate the average value between it and the corresponding second new facet map, and we can get Fig. 9 The third new bin diagram is shown.

[0193] For the initial bin map supported by the second new bin (ie, the supporting bin map), the average value between the two can be calculated to obtain the third new bin map. Exemplarily, the average value can include but is not limited to: arithmetic mean, geometric mean, weighted mean, etc.

[0194] Taking the weighted average as an example, the following formula can be used to fuse the initial surfel map supported by the second new surfel map with the second new surfel map:

[0195]

[0196]

[0197] σ s :=min{σ s +w,σ max},

[0198] Among them, p is the coordinate, c is the color, n is the normal vector, s is the face element, m is the pixel measurement, w is the inverse of the number of supported face elements, ":=" means "assignment", that is, the newly defined symbol on the right is defined to express the value on the left, withf(Vs, Vm) refers to the third new face element map, f(Ps, Pm) refers to substituting Ps and Pm into the right side of f(Vs, Vm), replacing Vs with Ps, and replacing Vm with Pm, f(ns, nm) refers to substituting ns and nm into the right side of f(Vs, Vm), replacing Vs with ns, and replacing Vm with nm, f(Cs, Cm) refers to substituting Cs and Cm into the right side of f(Vs, Vm), replacing Vs with Cs, and replacing Vm with Cm, and σmax refers to the confidence with the largest confidence among all face elements.

[0199] In the disclosed embodiment, for the old facet map supported by the second new facet map, the average value between the second new facet map and the old facet map can be calculated, so that the new and old facet maps of the same pixel point can be fused to generate a dense surface, thereby improving the accuracy of determining the facets of the same pixel point, reducing fusion noise, and thus improving the accuracy of three-dimensional reconstruction; in addition, for the supported old facet map, the fusion method corresponding to the supported old facet map is adopted for fusion, thereby improving the flexibility of facet fusion.

[0200] In an exemplary embodiment, in the above step S170333, the above merging the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map may include:

[0201] When the position relationship indicates that the distance between the initial surfel map and the second new surfel map is greater than a distance threshold, the initial surfel map is replaced based on the second new surfel map to obtain the third new surfel map.

[0202] In this embodiment, if the initial surfel image conflicts with the second new surfel image, the initial surfel image can be considered as an erroneous surfel image, and the initial surfel image can be replaced to obtain the third new surfel image, that is, the initial surfel image is directly deleted. Fig. 9 As shown, for the conflict surface element diagram ( Figure 8 The code "4" in the figure) can be used to replace the conflicting face element diagram with the corresponding second new face element diagram, and the Fig. 9 The third new bin diagram is shown.

[0203] In the disclosed embodiment, for an old facet map that conflicts with the second new facet map, the old facet map can be directly replaced, and the old and new facet maps of the same pixel can be fused to generate a dense surface, thereby improving the determination accuracy of the facet map of the same pixel, reducing fusion noise, and thus improving the accuracy of three-dimensional reconstruction; in addition, for the conflicting old facet map, the fusion method corresponding to the conflicting old facet map is adopted for fusion, thereby improving the flexibility of facet fusion.

[0204] In an exemplary embodiment, in the above step S170333, the above merging the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map may include:

[0205] When the positional relationship indicates that the initial surfel map is blocked by the second new surfel map, a new surfel map of the initial surfel map is created, and the created new surfel map and the second new surfel map are used as the third new surfel map.

[0206] In this embodiment, if the initial bin map is blocked by the second new bin map, a new surface may be created for the blocked old bin to obtain the third new bin map.

[0207] In the disclosed embodiment, a new surface is created for the old facet map occluded by the second new facet map, so that the old and new facet maps of the same pixel point can be fused to generate a dense surface, thereby improving the determination accuracy of the facet map of the same pixel point, reducing fusion noise, and thus improving the accuracy of three-dimensional reconstruction; in addition, for the occluded old facet map, the fusion method corresponding to the occluded old facet map is adopted for fusion, thereby improving the flexibility of facet fusion.

[0208] In step S19, a three-dimensional reconstruction result of the scene to be reconstructed is generated according to the surfel map corresponding to the first frame image and the fused surfel map; the surfel map corresponding to the first frame image is generated based on the point cloud data of the first frame image.

[0209] In the disclosed embodiment, three-dimensional reconstruction can be performed on the object to be reconstructed based on the facet map corresponding to the first frame image and the fused facet map of images other than the first frame image in the image sequence to obtain a three-dimensional reconstruction result of the scene to be reconstructed.

[0210] Fig.10 FIG. 1 is a flowchart of semantic fusion according to an exemplary embodiment. Fig.10 As shown, in an optional embodiment, the above method may further include:

[0211] In step S21, the facet map corresponding to the first frame image and the fused facet map are fused with the semantic segmentation result to obtain the semantically fused facet map corresponding to the image in the image sequence; the semantic segmentation result is obtained by semantic segmentation of map points, the map points are three-dimensional points that give depth information to feature points in the key frame image, the key frame image is an image in the historical frame image that partially overlaps with the current frame image, and the feature points are obtained by feature extraction of the key frame image.

[0212] In step S23, the above semantically fused facet graph is gridded to obtain the above three-dimensional reconstruction result.

[0213] In an exemplary embodiment, map points in the local map corresponding to the image sequence can be obtained in advance. The map points refer to the three-dimensional points after the depth is given to each feature point in the key frame image. The feature points are obtained by extracting features from the key frame image. And the map points are input into Figure 2 The preset deep network model shown is used to perform semantic segmentation on the map point to obtain a semantic segmentation result. Exemplarily, the semantic segmentation result may include multiple semantic labels and corresponding probability information. That is, the semantic segmentation may add a semantic label corresponding to the object category to each type of object in the local map, as well as the probability that each type of object belongs to the label.

[0214] In a feasible embodiment, in step S21, the initial facet map corresponding to the first frame image and the fused facet map corresponding to each frame image in the above-mentioned image sequence to be processed can be fused with the semantic segmentation result to obtain the semantically fused facet map corresponding to each frame image. Optionally, since each frame image usually contains multiple facet maps and there are generally multiple semantic labels, each facet in each frame image can be fused with one of the semantic labels to obtain the fused facet map corresponding to each frame image. The fusion formula can be as follows:

[0215]

[0216] Among them, I represents each frame image, l represents the semantic label value, z is a constant, P represents the probability value, s represents the face element, k represents the image frame, and P(l i |I 1，...，k-1 ) represents the prior probability value calculated from the 1st to K-1th frame image, P(O u(s，k) = l i |I k ) represents the posterior probability value of the first to K frames, P(l i |I 1，...，k ) represents the probability value calculated in the 1st to K-1th frame images.

[0217] Based on the above formula, the above step S21 can be understood as: taking the product of the prior probability value calculated in the 1st to K-1st frame image, the posterior probability value of the 1st to K-1st frame image and the constant z as the probability value calculated in the 1st to K-1st frame image. Among them, the prior probability value can be understood as the prior probability value corresponding to the semantically fused face element map predicted after the face element in the 1st to K-1st frame image is fused with the semantic label value; the posterior probability value can be understood as the posterior probability value corresponding to the semantically fused face element map obtained after the face element in the 1st to K-1st frame image is fused with the semantic label value.

[0218] In a feasible embodiment, in the above step S23, a triangulation algorithm (Delaunay) may be used to perform grid processing on the semantically fused facet graph to obtain the above three-dimensional reconstruction result.

[0219] In the disclosed embodiment, by fusing the fused facet map corresponding to each frame image with the semantic segmentation result, different facets in each frame image can store different semantic information and are rasterized to generate dense surfaces respectively, which can improve the accuracy of three-dimensional reconstruction.

[0220] Fig.11 is a block diagram of a 3D reconstruction device according to an exemplary embodiment. Fig.11 The device may include an image sequence acquisition module 31, a point cloud data generation module 33, an initial surface element map generation module 35, a fused surface element map generation module 37 and a three-dimensional reconstruction result generation module 39.

[0221] The image sequence acquisition module 31 is configured to execute the acquisition of an image sequence of the scene to be reconstructed.

[0222] The point cloud data generating module 33 is configured to determine the point cloud data of the images in the above-mentioned image sequence.

[0223] The initial facet map generation module 35 is configured to execute point cloud data of the historical frame image corresponding to the current frame image in the above-mentioned image sequence, and generate an initial facet map corresponding to the above-mentioned historical frame image; the above-mentioned current frame image is any frame image in the above-mentioned image sequence except the first frame image, and the above-mentioned historical frame image is an image in the above-mentioned image sequence whose acquisition time point is before the above-mentioned current frame image.

[0224] The fused surfel map generating module 37 is configured to generate a fused surfel map corresponding to the current frame image based on the initial surfel map and the point cloud data of the current frame image.

[0225] The three-dimensional reconstruction result generation module 39 is configured to generate a three-dimensional reconstruction result of the scene to be reconstructed based on the facet map corresponding to the first frame image and the fused facet map; the facet map corresponding to the first frame image is generated based on the point cloud data of the first frame image.

[0226] In an optional embodiment, the above-mentioned fused facet graph generation module includes:

[0227] The projection submodule is configured to project the initial surface element map onto the current frame image to obtain the projection pixel points of the initial surface element map in the current frame image.

[0228] The generation submodule is configured to generate the fused surface element map based on the projected pixel points and the point cloud data of the current frame image.

[0229] In an optional embodiment, the above-mentioned generation submodule includes:

[0230] The conversion unit is configured to perform a three-dimensional conversion on the coordinates of the projection pixel points and the coordinates of the pixel points in the current frame image based on the point cloud data of the current frame image, so as to obtain a first new facet map corresponding to the pixel points in the current frame image and a second new facet map corresponding to the projection pixel points.

[0231] The third new surfel map determining unit is configured to perform fusion of the initial surfel map and the second new surfel map to obtain a third new surfel map corresponding to the current frame image.

[0232] The fused surfel map determining unit is configured to merge the first new surfel map and the third new surfel map to obtain a fused surfel map corresponding to the current frame.

[0233] In an optional embodiment, the third new bin map determining unit includes:

[0234] The position relationship determination subunit is configured to determine the position relationship between the initial surfel map and the second new surfel map based on the depth information of the pixel points in the historical frame image and the depth information of the pixel points in the current frame image.

[0235] The fusion subunit is configured to fuse the initial surfel map and the second new surfel map according to the positional relationship to obtain the third new surfel map.

[0236] In an optional embodiment, the fusion subunit is configured to calculate the average value of the initial facet map and the second new facet map when the position relationship indicates that the distance between the initial facet map and the second new facet map is less than or equal to a distance threshold, so as to obtain the third new facet map.

[0237] In an optional embodiment, the fusion subunit is configured to replace the initial surfel map based on the second new surfel map to obtain the third new surfel map when the position relationship indicates that the distance between the initial surfel map and the second new surfel map is greater than a distance threshold.

[0238] In an optional embodiment, the fusion subunit is configured to create a new facet map of the initial facet map when the positional relationship indicates that the initial facet map is occluded by the second new facet map, and use the created new facet map and the second new facet map as the third new facet map.

[0239] In an optional embodiment, the above device further includes:

[0240] The semantic fusion module is configured to execute the fusion of the facet map corresponding to the above-mentioned first frame image and the above-mentioned fused facet map with the semantic segmentation result to obtain the semantically fused facet map corresponding to the image in the above-mentioned image sequence; the above-mentioned semantic segmentation result is obtained by semantic segmentation of map points, the above-mentioned map points are three-dimensional points that give depth information to feature points in the key frame image, the above-mentioned key frame image is an image in the above-mentioned historical frame image that partially overlaps with the above-mentioned current frame image, and the above-mentioned feature points are obtained by feature extraction of the above-mentioned key frame image.

[0241] The gridding processing module is configured to perform gridding processing on the facet image after the semantic fusion to obtain the three-dimensional reconstruction result.

[0242] In an optional embodiment, the initial bin graph generation module includes:

[0243] The pixel facet map generation unit is configured to execute a three-dimensional transformation of the coordinates of the pixel points in the historical frame image based on the point cloud data of the historical frame image to obtain a facet map corresponding to the pixel points in the historical frame image.

[0244] The initial surfel map determining unit is configured to execute the surfel map corresponding to the pixel point as the initial surfel map.

[0245] In an optional embodiment, the point cloud data generating module includes:

[0246] The depth and posture determination unit is configured to determine the depth information of the pixel points of the images in the above image sequence and the posture information of the images in the above image sequence.

[0247] The point cloud data generating unit is configured to generate point cloud data of the images in the above image sequence based on the depth information of the pixel points of the above image and the above posture information.

[0248] In an optional embodiment, the above-mentioned posture information includes rotation posture information and translation posture information, and the above-mentioned device includes:

[0249] The rotation posture information acquisition module is configured to acquire the world coordinate information of the feature points in the previous frame image and the rotation posture information of the current frame image; the previous frame image is an adjacent image in the image sequence, whose acquisition time point is before the current frame image; the feature points in the previous frame image are obtained by feature extraction of the previous frame image.

[0250] The optical flow tracking module is configured to perform optical flow tracking on the feature points in the above-mentioned previous frame image to determine the image coordinate information of the feature points in the above-mentioned current frame image; the feature points in the above-mentioned current frame image are obtained by feature extraction of the above-mentioned current frame image.

[0251] The translation posture information determination module is configured to determine the translation posture information of the current frame image based on the world coordinate information, the image coordinate information and the rotation posture information of the current frame image.

[0252] In an optional embodiment, the apparatus further includes an adjustment module for adjusting the translation posture information, and the adjustment module includes:

[0253] The key frame image acquisition unit is configured to acquire a key frame image from the historical frame image; the key frame image partially overlaps with the current frame image.

[0254] The local map construction unit is configured to construct a local map using feature points in the key frame image; the feature points in the key frame image are obtained by extracting features from the key frame image.

[0255] The matching unit is configured to perform matching of pixel points in the current frame image based on the local map.

[0256] The posture adjustment unit is configured to adjust the above-mentioned translation posture information based on the matching result to obtain the adjusted translation posture information of the above-mentioned current frame image.

[0257] In an optional embodiment, the point cloud data generating unit includes:

[0258] The first position information acquisition subunit is configured to acquire the first position information of the image collector corresponding to the first frame image.

[0259] The second position information determination subunit is configured to determine the second position information of the image collector corresponding to any of the above-mentioned frame images based on the depth information, posture information and the above-mentioned first position information of the pixel points in any of the above-mentioned images.

[0260] The world coordinate information determination subunit is configured to determine the world coordinate information of the pixel points in the above image based on the position information corresponding to the above image and the depth information of the pixel points in the above image; the position information corresponding to the above image includes the first position information and the second position information.

[0261] The point cloud data generating subunit is configured to generate point cloud data of the image according to the world coordinate information of the pixel points in the image.

[0262] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0263] In an exemplary embodiment, an electronic device for three-dimensional reconstruction is also provided, including a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps of any three-dimensional reconstruction method in the above-mentioned embodiments when executing the instructions stored in the memory.

[0264] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.12It is a block diagram of an electronic device for three-dimensional reconstruction according to an exemplary embodiment. The electronic device 40 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 41 (the central processing unit 41 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 43 for storing data, and one or more storage media 42 (such as one or more mass storage devices) for storing application programs 423 or data 422. Among them, the memory 43 and the storage medium 42 can be short-term storage or permanent storage. The program stored in the storage medium 42 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 41 may be configured to communicate with the storage medium 42 to execute a series of instruction operations in the storage medium 42 on the electronic device 40. The electronic device 40 may also include one or more power supplies 46, one or more wired or wireless network interfaces 45, one or more input and output interfaces 44, and / or one or more operating systems 421, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0265] The input / output interface 44 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 40. In one example, the input / output interface 44 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In an exemplary embodiment, the input / output interface 44 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0266] It can be understood by those skilled in the art that Fig.12 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.12 More or fewer components as shown, or with Fig.12 Different configurations shown.

[0267] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the steps of any three-dimensional reconstruction method in the above embodiments.

[0268] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the three-dimensional reconstruction method provided in any one of the above embodiments is implemented.

[0269] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0270] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0271] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire an image sequence of a scene to be reconstructed; determining point cloud data of images in the image sequence; Based on the point cloud data of the historical frame image corresponding to the current frame image in the image sequence, an initial facet map corresponding to the historical frame image is generated; the current frame image is any frame image except the first frame image in the image sequence, and the historical frame image is an image in the image sequence whose acquisition time point is before the current frame image; wherein the facet map refers to replacing the object surface of the scene to be reconstructed with a plane or a curved surface; Projecting the initial facet map onto the current frame image to obtain the projected pixel points of the initial facet map in the current frame image; performing a three-dimensional transformation on the coordinates of the projected pixel points and the coordinates of the pixel points in the current frame image based on the point cloud data of the current frame image to obtain a first new facet map corresponding to the pixel points in the current frame image and a second new facet map corresponding to the projected pixel points; fusing the initial facet map and the second new facet map to obtain a third new facet map corresponding to the current frame image; merging the first new facet map and the third new facet map to obtain a fused facet map corresponding to the current frame image; A three-dimensional reconstruction result of the scene to be reconstructed is generated according to the facet map corresponding to the first frame image and the fused facet map; the facet map corresponding to the first frame image is generated based on the point cloud data of the first frame image.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of fusing the initial surfel map and the second new surfel map to obtain a third new surfel map corresponding to the current frame image includes: Determining a positional relationship between the initial surfel map and the second new surfel map according to depth information of pixels in the historical frame image and depth information of pixels in the current frame image; According to the positional relationship, the initial surfel map and the second new surfel map are fused to obtain the third new surfel map.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The step of fusing the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map comprises: When the position relationship indicates that the distance between the initial surfel map and the second new surfel map is less than or equal to a distance threshold, an average value of the initial surfel map and the second new surfel map is calculated to obtain the third new surfel map.

4. The three-dimensional reconstruction method according to claim 2, characterized in that: The step of fusing the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map comprises: In a case where the position relationship is used to indicate that the distance between the initial surfel map and the second new surfel map is greater than a distance threshold, the initial surfel map is replaced based on the second new surfel map to obtain the third new surfel map.

5. The three-dimensional reconstruction method according to claim 2, characterized in that: The step of fusing the initial surfel map and the second new surfel map according to the position relationship to obtain the third new surfel map comprises: When the positional relationship indicates that the initial surfel map is blocked by the second new surfel map, a new surfel map of the initial surfel map is created, and the created new surfel map and the second new surfel map are used as the third new surfel map.

6. The three-dimensional reconstruction method according to claim 1, characterized in that: The method further comprises: The facet map corresponding to the first frame image and the fused facet map are fused with the semantic segmentation result to obtain the semantically fused facet map corresponding to the image in the image sequence; the semantic segmentation result is obtained by semantically segmenting the map points, the map points are three-dimensional points that give depth information to the feature points in the key frame image, the key frame image is an image in the historical frame image that partially overlaps with the current frame image, and the feature points are obtained by feature extraction of the key frame image; The semantically fused facet graph is subjected to gridding processing to obtain the three-dimensional reconstruction result.

7. The three-dimensional reconstruction method according to any one of claims 1 to 6, characterized in that: The step of generating an initial facet map corresponding to a historical frame image based on point cloud data of a historical frame image corresponding to a current frame image in the image sequence comprises: Based on the point cloud data of the historical frame image, coordinates of pixel points in the historical frame image are three-dimensionally transformed to obtain a facet map corresponding to the pixel points in the historical frame image; The bin map corresponding to the pixel point is used as the initial bin map.

8. The three-dimensional reconstruction method according to any one of claims 1 to 6, characterized in that: The determining of the point cloud data of the images in the image sequence comprises: Determining depth information of pixel points of images in the image sequence and position information of images in the image sequence; Point cloud data of the images in the image sequence are generated according to the depth information of the pixels of the image and the posture information.

9. The three-dimensional reconstruction method according to claim 8, characterized in that: The posture information includes rotation posture information and translation posture information, and the method includes: Obtaining world coordinate information of feature points in a previous frame image and rotational posture information of the current frame image; the previous frame image is an adjacent image in the image sequence whose acquisition time point is before the current frame image; the feature points in the previous frame image are obtained by extracting features from the previous frame image; Performing optical flow tracking on the feature points in the previous frame image to determine image coordinate information of the feature points in the current frame image; the feature points in the current frame image are obtained by performing feature extraction on the current frame image; Based on the world coordinate information, the image coordinate information and the rotational pose information of the current frame image, the translational pose information of the current frame image is determined.

10. The three-dimensional reconstruction method according to claim 9, characterized in that: The method further comprises a step of adjusting the translation posture information, wherein the step of adjusting the translation posture information comprises: Acquire a key frame image from the historical frame image; the key frame image partially overlaps with the current frame image; Constructing a local map using feature points in the key frame image; the feature points in the key frame image are obtained by extracting features from the key frame image; Matching pixels in the current frame image based on the local map; The translation pose information is adjusted based on the matching result to obtain the adjusted translation pose information of the current frame image.

11. The three-dimensional reconstruction method according to claim 8, characterized in that: The step of generating point cloud data of the images in the image sequence according to the depth information of the pixels of the images and the posture information comprises: Acquire first position information of the image collector corresponding to the first frame image; Determine the second position information of the image collector corresponding to any frame of image according to the depth information, the position information and the first position information of the pixel points in any image; Determine world coordinate information of a pixel point in the image according to the position information corresponding to the image and the depth information of the pixel point in the image; the position information corresponding to the image includes first position information and second position information; Point cloud data of the image is generated according to world coordinate information of pixel points in the image.

12. A three-dimensional reconstruction device, characterized in that: include: An image sequence acquisition module is configured to acquire an image sequence of a scene to be reconstructed; A point cloud data generation module, configured to determine point cloud data of an image in the image sequence; The initial facet map generation module is configured to generate an initial facet map corresponding to a historical frame image based on point cloud data of a historical frame image corresponding to a current frame image in the image sequence; the current frame image is any frame image except the first frame image in the image sequence, and the historical frame image is an image in the image sequence whose acquisition time point is before the current frame image; wherein the facet map refers to replacing an object surface of a scene to be reconstructed with a plane or a curved surface; The fused facet map generation module is configured to generate a fused facet map corresponding to the current frame image based on the initial facet map and the point cloud data of the current frame image; the fused facet map generation module includes: a projection submodule, configured to project the initial facet map into the current frame image to obtain the projection pixel points of the initial facet map in the current frame image; a generation submodule, configured to generate the fused facet map based on the projection pixel points and the point cloud data of the current frame image; the generation submodule includes: a conversion unit, configured to perform point cloud data of the current frame image, performing three-dimensional transformation on the coordinates of the projected pixel points and the coordinates of the pixel points in the current frame image to obtain a first new facet map corresponding to the pixel points in the current frame image and a second new facet map corresponding to the projected pixel points; a third new facet map determining unit, configured to fuse the initial facet map and the second new facet map to obtain a third new facet map corresponding to the current frame image; a fused facet map determining unit, configured to merge the first new facet map and the third new facet map to obtain a fused facet map corresponding to the current frame image; The three-dimensional reconstruction result generation module is configured to generate a three-dimensional reconstruction result of the scene to be reconstructed based on the facet map corresponding to the first frame image and the fused facet map; the facet map corresponding to the first frame image is generated based on the point cloud data of the first frame image.

13. An electronic device, characterized in that: include: processor; A memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the three-dimensional reconstruction method according to any one of claims 1 to 11.

14. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the three-dimensional reconstruction method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 11 is implemented.

Images