3D Model Generation Method, Device, Equipment and Medium Based on Image Sequence

Abstract

This application relates to the field of image processing technology, and discloses a three-dimensional model generation method, device, equipment and medium based on an image sequence. The method includes obtaining multiple images of an object from different angles as target images; through feature extraction and feature matching processing on the target images, obtaining sparse point cloud information of the object, and performing reconstruction processing on the sparse point cloud information of the object to obtain first sparse point cloud information; obtaining a segmentation mask of the target images, and performing foreground segmentation processing on the target images based on the segmentation mask to obtain second sparse point cloud information; performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain object region point cloud; through reconstructing the object region point cloud, obtaining a three-dimensional model of the object. By performing foreground segmentation processing on the target images, the present invention removes the cluttered background in the generated three-dimensional model and improves the accuracy of three-dimensional model generation.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a method, apparatus, device, and medium for generating a three-dimensional model based on an image sequence. Background Art

[0002] Constructing a 3D model from multi-view pictures of an object is a major research direction in computer graphics and computer vision, known as Image based modeling. This technology has gradually matured and entered the practical stage. Some software allows users to take several photos of the physical object to be modeled from different perspectives with an ordinary camera or mobile phone, and then the software can automatically generate the corresponding 3D model based on these photos.

[0003] However, existing three-dimensional modeling technologies based on image sequences will also reconstruct the background part of the photos that is not the object. This will result in the reconstructed model not being an independent and clean three-dimensional model of a single object, but a three-dimensional space mixed with the background, thereby causing it to be interfered by the object background information and making the accuracy of the generated three-dimensional model relatively low. Summary of the Invention

[0004] The purpose of the embodiments of this application is to propose a method, apparatus, device, and medium for generating a three-dimensional model based on an image sequence to improve the accuracy of three-dimensional model generation.

[0005] To solve the above technical problems, the embodiments of this application provide a method for generating a three-dimensional model based on an image sequence, including:

[0006] Obtain multiple images of an object from different angles as target images;

[0007] Through feature extraction and feature matching processing on the target images, obtain the sparse point cloud information of the object, and perform reconstruction processing on the sparse point cloud information of the object to obtain the first sparse point cloud information;

[0008] Obtain the segmentation mask of the target images, and perform foreground segmentation processing on the target images based on the segmentation mask to obtain the second sparse point cloud information;

[0009] Perform point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain the object region point cloud;

[0010] Through reconstruction processing on the object region point cloud, obtain the three-dimensional model of the object.

[0011] To solve the above technical problems, the embodiments of this application provide a device for generating a three-dimensional model based on an image sequence, including:

[0012] A target image acquisition module, configured to acquire multiple images of an object at different angles as target images;

[0013] A first sparse point cloud information acquisition module, configured to obtain object sparse point cloud information by performing feature extraction and feature matching processing on the target image, and perform reconstruction processing on the object sparse point cloud information to obtain first sparse point cloud information;

[0014] A second sparse point cloud information acquisition module, configured to obtain a segmentation mask of the target image, and perform foreground segmentation processing on the target image based on the segmentation mask to obtain second sparse point cloud information;

[0015] An object region point cloud generation module, configured to perform point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain object region point cloud;

[0016] A three-dimensional model generation module, configured to obtain a three-dimensional model of the object by performing reconstruction processing on the object region point cloud.

[0017] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that one or more processors implement the three-dimensional model generation method based on an image sequence described in any one of the above.

[0018] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the three-dimensional model generation method based on an image sequence described in any one of the above.

[0019] An embodiment of the present invention provides a three-dimensional model generation method, device, device and medium based on an image sequence. Among them, the method includes: acquiring multiple images of an object at different angles as target images; obtaining object sparse point cloud information by performing feature extraction and feature matching processing on the target image, and performing reconstruction processing on the object sparse point cloud information to obtain first sparse point cloud information; obtaining a segmentation mask of the target image, and performing foreground segmentation processing on the target image based on the segmentation mask to obtain second sparse point cloud information; performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain object region point cloud; obtaining a three-dimensional model of the object by performing reconstruction processing on the object region point cloud. The embodiment of the present invention performs foreground segmentation processing on the target image, removing the cluttered background in the generated three-dimensional model, and at the same time performing point cloud screening on the sparse point cloud, further reducing the invalid point cloud in the production of the three-dimensional model, thereby being beneficial to improving the accuracy of three-dimensional model generation. Description of the Drawings

[0020] To more clearly illustrate the solutions in this application, the following will give a brief introduction to the drawings required for the description of the embodiments of this application. Obviously, the drawings in the following description are some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of an implementation of the method for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0022] Figure 2 It is another flowchart of an implementation of a sub-process in the method for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0023] Figure 3 It is another flowchart of an implementation of a sub-process in the method for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0024] Figure 4 It is another flowchart of an implementation of a sub-process in the method for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0025] Figure 5 It is another flowchart of an implementation of a sub-process in the method for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0026] Figure 6 It is another flowchart of an implementation of a sub-process in the method for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0027] Figure 7 It is a schematic diagram of a device for generating a three-dimensional model based on an image sequence provided by an embodiment of this application;

[0028] Figure 8 It is a schematic diagram of a computer device provided by an embodiment of this application. Detailed implementation manners

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0030] References to "embodiments" in this specification mean that specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0032] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the method for generating a three-dimensional model based on an image sequence provided by the embodiments of the present application is generally executed by a server. Correspondingly, the device for generating a three-dimensional model based on an image sequence is generally configured in the server.

[0034] Please refer to Figure 1 , Figure 1 which shows a specific implementation of the method for generating a three-dimensional model based on an image sequence.

[0035] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 the process sequence shown, and the method includes the following steps:

[0036] S1: Obtain multiple images of the object from different angles as target images.

[0037] Specifically, when it is necessary to construct a three-dimensional model of an object, first obtain multiple images of the object from different angles. The images from different angles can be the front view, left view, right view, rear view, top view, and bottom view of the object.

[0038] S2: Through feature extraction and feature matching processing on the target images, obtain the sparse point cloud information of the object, and perform reconstruction processing on the sparse point cloud information of the object to obtain the first sparse point cloud information.

[0039] Specifically, since the target images are images of the object from different angles, each target image is subjected to feature extraction to obtain the image features of each target image; then the image features of different target images are subjected to feature matching, and the same features are combined to obtain different feature points, that is, the sparse point cloud information of the object, and then reconstruction processing is performed on the sparse point cloud information of the object to finally obtain the first sparse point cloud information.

[0040] Among them, the set of point data on the surface of the product appearance obtained by a measuring instrument in reverse engineering is also called point cloud. Usually, the number of points obtained by a three-dimensional coordinate measuring machine is relatively small, and the distance between points is relatively large, which is called sparse point cloud; while the point cloud obtained by using a three-dimensional laser scanner or a photogrammetric scanner has a relatively large and dense number of points, which is called dense point cloud or thick point cloud. In the embodiment of the present application, the first sparse point cloud information is a massive point set of surface features in the target image.

[0041] Please refer to Figure 2 , Figure 2 which shows a specific implementation manner of step S2, described in detail as follows:

[0042] S21: For each target image, image features of each target image are obtained by performing feature extraction of scale-invariant feature transform on the target image.

[0043] S22: The image features of multiple target images are subjected to feature matching processing to obtain object sparse point cloud information.

[0044] S23: An incremental structure from motion recovery algorithm is used to reconstruct the object sparse point cloud information to obtain the first sparse point cloud information.

[0045] Specifically, scale-invariant feature transform, that is, SIFT (Scale-invariant feature transform), is a description used in the field of image processing. This description has scale invariance, can detect key points in the image, and is a local feature descriptor. In the embodiment of the present application, since the target images are images at different angles, in order to maintain the scale invariance of the images at different angles, feature extraction of scale-invariant feature transform is performed on the target images to obtain the image features of each target image. Then, the image features of different target images are subjected to feature matching, and the same features are structured to obtain different feature points, that is, object sparse point cloud information. Finally, an incremental structure from motion recovery algorithm is used to reconstruct the object sparse point cloud information to obtain the first sparse point cloud information.

[0046] Among them, the incremental structure from motion recovery algorithm refers to incremental SfM (Structure from Motion); and the SFM algorithm is an offline algorithm for three-dimensional reconstruction based on various collected unordered pictures. In the embodiment of the present application, an incremental structure from motion recovery algorithm is used to reconstruct the object sparse point cloud information to obtain the first sparse point cloud information.

[0047] In this example, for each target image, image features of each target image are obtained by performing feature extraction of scale-invariant feature transform on the target image, and the image features of multiple target images are subjected to feature matching processing to obtain object sparse point cloud information. Then, an incremental structure-from-motion algorithm is used to reconstruct the object sparse point cloud information to obtain the first sparse point cloud information, realizing feature recognition and matching of the target image and obtaining sparse point cloud information, which is beneficial to extracting point cloud information for the three-dimensional model of the object and providing a basis for generating the three-dimensional model of the object.

[0048] Please refer to Figure 3 , Figure 3 which shows a specific implementation manner of step S21, described in detail as follows:

[0049] S211: For each target image, extract the key points of the target image to obtain target key points;

[0050] S212: By positioning the target key points and determining the feature directions, obtain the feature vectors of the key points;

[0051] S213: Match the feature vectors of the key points to perform feature extraction of scale-invariant feature transform on the target image to obtain the image features of each target image.

[0052] Specifically, for each image, extract key points. These key points are some very prominent points in the target image that will not disappear due to factors such as illumination, scale, and rotation, such as corner points, edge points, bright points in dark regions, and dark points in bright regions. Extracting key points is to search for image positions in all scale spaces and then identify potential interest points with scale and rotation invariance through Gaussian differential functions.

[0053] At the position of each candidate key point, determine the position and scale through a finely fitted model; the selection of these key points is based on their stability; then, based on the local gradient direction of the image, assign one or more directions to each key point position. All subsequent operations on the image data are transformed relative to the direction, scale, and position of the key points, thereby providing invariance to these transformations, that is, by positioning the target key points and determining the feature directions, obtain the feature vectors of the key points. Finally, through the feature vectors of each key point, compare them pairwise to find several pairs of matching feature points, establish the corresponding relationship between objects, and finally obtain the image features of each target image.

[0054] In this embodiment, for each target image, key points of the target image are extracted to obtain target key points, and by positioning the target key points and determining the feature direction, feature vectors of the key points are obtained. Finally, the feature vectors of the key points are matched to perform scale-invariant feature transform feature extraction on the target image, obtain the image features of each target image, and achieve feature extraction of the target image, providing a basis for subsequent three-dimensional map model generation.

[0055] S3: Obtain the segmentation mask of the target image, and perform foreground segmentation processing on the target image based on the segmentation mask to obtain the second sparse point cloud information.

[0056] Specifically, the segmentation mask, also known as the image mask, refers to using a selected image, graphic, or object to occlude the processed image (entirely or partially) to control the area or process of image processing. The specific image or object used for covering is called a mask or template. In the embodiment of the present application, by identifying the segmentation mask of the target image, the object is segmented from the target image, thereby removing the cluttered background in the generation of the object three-dimensional map.

[0057] Please refer to Figure 4 , Figure 4 which shows a specific implementation manner of step S3, described in detail as follows:

[0058] S31: Use a foreground segmentation algorithm to identify the object information in the target image to obtain the segmentation mask.

[0059] S32: Based on the segmentation mask, perform foreground segmentation processing on the object from the target image to obtain the second sparse point cloud information.

[0060] Specifically, in the embodiments of the present application, the foreground segmentation algorithm adopted is the U-Net algorithm. This algorithm is a semantic segmentation algorithm that uses a fully convolutional neural network. First, it performs convolutional processing and pooling (feature extraction layer) through a feature extraction network. Then, the feature fusion network uses the feature map generated by upsampling to perform a concatenate operation with the feature map generated by the feature extraction network. Finally, after two more convolutional operations, a feature map is generated, and two convolutions with a kernel size of 1*1 are used for classification to obtain the final two heatmaps. For example, the first one represents the score of the first category, and the second one represents the score heatmap of the second category. Then, as the input of the softmax function, the softmax with a relatively large probability is calculated, and then loss calculation and backpropagation calculation are performed to finally achieve foreground segmentation. In this embodiment, the U-Net algorithm is used to identify the object information in the target image to obtain a segmentation mask, and then based on the segmentation mask, the object is foreground-segmented from the target image to obtain the second sparse point cloud information, effectively removing the cluttered background in the three-dimensional object graph model and greatly reducing the number of point clouds that need to be depth-estimated, which is beneficial to improving the accuracy and efficiency of the generation of the three-dimensional object graph model.

[0061] S4: Perform point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain the point cloud of the object region.

[0062] Please refer to Figure 5 , Figure 5 which shows a specific implementation manner of step S4, described in detail as follows:

[0063] S41: Obtain the mapping relationship of the point clouds in the first sparse point cloud information and the second sparse point cloud information, where the mapping relationship is the mapping connection between the two-dimensional object point cloud and the three-dimensional object point cloud.

[0064] S42: Based on the mapping relationship, perform point cloud screening on the first sparse point cloud information and the second sparse point cloud information to extract the point cloud of the object region in the target image and obtain the point cloud of the object region.

[0065] Specifically, since each point cloud information in the target image is correspondingly extracted during the process of extracting the sparse point cloud information of the target image, during the process of obtaining the first sparse point cloud information and the second sparse point cloud information, it includes the mapping relationship between the two-dimensional object point cloud and the three-dimensional object point cloud; and since the second sparse point cloud information is obtained by performing foreground segmentation on the target image and mainly includes the point cloud information of the object area, by obtaining the mapping relationship of the point clouds in the first sparse point cloud information and the second sparse point cloud information, and then based on the mapping relationship, screening the point clouds of the first sparse point cloud information and the second sparse point cloud information to extract the point clouds of the object area in the target image, obtaining the point clouds of the object area, the point clouds of the cluttered background are removed, and the number of point clouds other than the subsequent depth estimation is retrieved, which is beneficial to improving the accuracy and efficiency of the generation of the three-dimensional object map model.

[0066] S5: Reconstruct the point clouds of the object area to obtain a three-dimensional model of the object.

[0067] Specifically, the point clouds of the object area in the target image have been obtained in the above steps, and then the point clouds of the object area are reconstructed to obtain a three-dimensional model of the object. The reconstruction process includes depth map estimation processing, meshing, and texture mapping processing.

[0068] The embodiment of the present invention provides a three-dimensional model generation method, device, equipment, and medium based on an image sequence. Among them, the method includes: obtaining multiple different-angle images of an object as target images; through feature extraction and feature matching processing on the target images, obtaining object sparse point cloud information, and performing reconstruction processing on the object sparse point cloud information to obtain first sparse point cloud information; obtaining a segmentation mask of the target image, and performing foreground segmentation processing on the target image based on the segmentation mask to obtain second sparse point cloud information; screening the point clouds of the first sparse point cloud information and the second sparse point cloud information to obtain point clouds of the object area; reconstructing the point clouds of the object area to obtain a three-dimensional model of the object. By performing foreground segmentation processing on the target image in the embodiment of the present invention, the cluttered background in the generated three-dimensional model is removed, and at the same time, the sparse point clouds are screened, further reducing the invalid point clouds in the production of the three-dimensional model, thereby being beneficial to improving the accuracy of the three-dimensional model generation.

[0069] Please refer to Figure 6 , Figure 6 which shows a specific implementation manner of step S5, described in detail as follows:

[0070] S51: Use the semi-global matching algorithm to perform depth map estimation processing on the point clouds of the object area to obtain dense point clouds of the object.

[0071] S52: The three-dimensional model of the object is obtained by performing meshing and texture mapping on the dense point cloud of the object.

[0072] Specifically, the semi-global matching algorithm refers to the SGM (semi-global matching) algorithm, which is a semi-global matching algorithm for calculating disparity in binocular vision. Its implementation in OpenCV is semi-global blockmatching (SGBM). The idea of SGBM is as follows: by selecting the disparity of each pixel point to form a disparity map, a global energy function related to the disparity map is set, and this energy function is minimized to achieve the purpose of solving the optimal disparity of each pixel. In the embodiments of the present application, the semi-global matching algorithm is adopted to perform depth map estimation processing on the point cloud of the object region to obtain the dense point cloud of the object. Among them, depth estimation processing is to estimate the distance of each pixel in the image relative to the shooting source by using one or a unique / multiple perspective RGB images. In the embodiments of the present application, depth map estimation processing is to re-estimate and calculate the point cloud of the object region to obtain the dense point cloud of the object. Among them, texture mapping is the process of mapping texture pixels in the texture space to pixels in the screen space.

[0073] In this embodiment, the semi-global matching algorithm is adopted to perform depth map estimation processing on the point cloud of the object region to obtain the dense point cloud of the object, and then the three-dimensional model of the object is obtained by performing meshing and texture mapping on the dense point cloud of the object, realizing the generation of the three-dimensional map of the object, removing the cluttered background in the generated three-dimensional model, and at the same time performing point cloud screening on the sparse point cloud to further reduce the invalid point cloud in the production of the three-dimensional model, which is beneficial to improving the accuracy of three-dimensional model generation.

[0074] Please refer to Figure 7 , as an implementation of the above Figure 1 shown method, an embodiment of a three-dimensional model generation device based on an image sequence is provided in the present application. This device embodiment corresponds to the Figure 1 shown method embodiment, and this device can be specifically applied to various electronic devices.

[0075] As Figure 7 shown, the three-dimensional model generation device based on an image sequence in this embodiment includes: a target image acquisition module 61, a first sparse point cloud information acquisition module 62, a second sparse point cloud information acquisition module 63, an object region point cloud generation module 64, and a three-dimensional map model generation module 65, where:

[0076] A target image acquisition module 61, configured to acquire multiple images of an object at different angles as target images;

[0077] A first sparse point cloud information acquisition module 62, configured to perform feature extraction and feature matching processing on the target images to obtain object sparse point cloud information, and perform reconstruction processing on the object sparse point cloud information to obtain first sparse point cloud information;

[0078] A second sparse point cloud information acquisition module 63, configured to acquire a segmentation mask of the target images, and perform foreground segmentation processing on the target images based on the segmentation mask to obtain second sparse point cloud information;

[0079] An object region point cloud generation module 64, configured to perform point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain object region point cloud;

[0080] A three-dimensional model generation module 65, configured to perform reconstruction processing on the object region point cloud to obtain a three-dimensional model of the object.

[0081] Furthermore, the first sparse point cloud information acquisition module 62 includes:

[0082] An image feature extraction unit, configured to perform feature extraction of scale-invariant feature transform on each target image to obtain image features of each target image;

[0083] An object sparse point cloud information acquisition unit, configured to perform feature matching processing on the image features of multiple target images to obtain object sparse point cloud information;

[0084] An object sparse point cloud information reconstruction unit, configured to perform reconstruction processing on the object sparse point cloud information by using an incremental structure from motion algorithm to obtain first sparse point cloud information.

[0085] Furthermore, the image feature extraction unit includes:

[0086] A target key point extraction subunit, configured to extract key points of each target image to obtain target key points;

[0087] A feature vector acquisition subunit, configured to obtain a feature vector of a key point by positioning the target key point and determining a feature direction;

[0088] A feature vector matching subunit, configured to match the feature vectors of key points so as to perform feature extraction of scale-invariant feature transform on the target images to obtain image features of each target image.

[0089] Furthermore, the second sparse point cloud information acquisition module 63 includes:

[0090] A segmentation mask acquisition unit, which is used to identify object information in a target image by using a foreground segmentation algorithm to obtain a segmentation mask;

[0091] A foreground segmentation unit, which is used to perform foreground segmentation processing on an object from the target image based on the segmentation mask to obtain second sparse point cloud information.

[0092] Further, the object region point cloud generation module 64 includes:

[0093] A mapping relationship acquisition unit, which is used to obtain the mapping relationship of the point clouds in the first sparse point cloud information and the second sparse point cloud information, where the mapping relationship is the mapping connection between the two-dimensional object point cloud and the three-dimensional object point cloud;

[0094] A point cloud screening unit, which is used to screen the point clouds of the first sparse point cloud information and the second sparse point cloud information based on the mapping relationship to extract the point clouds of the object region in the target image to obtain the object region point cloud.

[0095] Further, the three-dimensional map model generation module 65 includes:

[0096] A dense point cloud generation unit, which is used to perform depth map estimation processing on the object region point cloud by using a semi-global matching algorithm to obtain an object dense point cloud;

[0097] A texture mapping unit, which is used to obtain a three-dimensional map model of the object by performing meshing and texture mapping processing on the object dense point cloud.

[0098] To solve the above technical problems, the embodiments of the present application also provide a computer device. For details, please refer to Figure 8 , Figure 8 which is the basic structural block diagram of the computer device in this embodiment.

[0099] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that communicate with each other through a system bus. It should be noted that only a computer device 7 with three components, namely a memory 71, a processor 72, and a network interface 73, is shown in the figure. However, it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art of the present technology can understand that a computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0100] The computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, etc.

[0101] The memory 71 includes at least one type of readable storage medium. The readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit and the external storage device of the computer device 7. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as the program code of the three-dimensional model generation method based on an image sequence. In addition, the memory 71 can also be used to temporarily store various data that have been output or will be output.

[0102] In some embodiments, the processor 72 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run the program code stored in the memory 71 or process data, such as running the program code of the above-mentioned three-dimensional model generation method based on an image sequence to implement various embodiments of the three-dimensional model generation method based on an image sequence.

[0103] The network interface 73 can include a wireless network interface or a wired network interface. The network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0104] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to execute the steps of a three-dimensional model generation method based on an image sequence as described above.

[0105] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of various embodiments of the present application.

[0106] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, a string of data blocks generated by using cryptographic methods. Each data block contains information on a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0107] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields is equally within the scope of the patent protection of the present application.

Claims

1. A method for generating a three-dimensional model based on an image sequence, characterized in that Including: Obtaining multiple images of an object from different angles as target images; Through feature extraction and feature matching processing on the target images, obtaining object sparse point cloud information, and performing reconstruction processing on the object sparse point cloud information to obtain first sparse point cloud information; Obtaining a segmentation mask of the target images, and performing foreground segmentation processing on the target images based on the segmentation mask to obtain second sparse point cloud information; Performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain object region point cloud; Through reconstructing the object region point cloud, obtaining a 3D model of the object; The performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain object region point cloud includes: Obtaining the mapping relationship of the point clouds in the first sparse point cloud information and the second sparse point cloud information, where the mapping relationship is the mapping connection between 2D object point cloud and 3D object point cloud; Based on the mapping relationship, performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information to extract the point cloud of the object region in the target images, obtaining the object region point cloud.

2. The three-dimensional model generation method based on an image sequence according to claim 1, characterized in that The through feature extraction and feature matching processing on the target images, obtaining object sparse point cloud information, and performing reconstruction processing on the object sparse point cloud information to obtain first sparse point cloud information includes: For each of the target images, through performing feature extraction of scale-invariant feature transform on the target image, obtaining the image features of each of the target images; Performing feature matching processing on the image features of multiple target images to obtain the object sparse point cloud information; Using an incremental structure-from-motion algorithm to perform reconstruction processing on the object sparse point cloud information to obtain the first sparse point cloud information.

3. The three-dimensional model generation method based on an image sequence according to claim 2, wherein The for each of the target images, through performing feature extraction of scale-invariant feature transform on the target image, obtaining the image features of each of the target images includes: For each of the target images, extracting the key points of the target image to obtain target key points; Through positioning the target key points and determining the feature directions, obtaining the feature vectors of the key points; Matching the feature vectors of the key points to perform feature extraction of scale-invariant feature transform on the target image, obtaining the image features of each of the target images.

4. The 3D model generation method based on an image sequence according to claim 1, characterized in that, The obtaining the segmentation mask of the target images, and performing foreground segmentation processing on the target images based on the segmentation mask to obtain second sparse point cloud information includes: Using a foreground segmentation algorithm to identify the object information in the target images to obtain the segmentation mask; Based on the segmentation mask, performing foreground segmentation processing on the object from the target images to obtain the second sparse point cloud information.

5. The method for generating a three-dimensional model based on an image sequence according to any one of claims 1 to 4, characterized in that, The through reconstructing the object region point cloud, obtaining a 3D model of the object includes: Using a semi-global matching algorithm to perform depth map estimation processing on the object region point cloud to obtain object dense point cloud; By performing meshing and texture mapping on the dense point cloud of the object, a 3D model of the object is obtained.

6. A three-dimensional model generation device based on an image sequence, characterized in that, It includes: A target image acquisition module for acquiring multiple images of the object from different angles as target images; A first sparse point cloud information acquisition module for obtaining object sparse point cloud information by performing feature extraction and feature matching on the target images, and performing reconstruction processing on the object sparse point cloud information to obtain the first sparse point cloud information; A second sparse point cloud information acquisition module for obtaining a segmentation mask of the target image and performing foreground segmentation on the target image based on the segmentation mask to obtain the second sparse point cloud information; An object region point cloud generation module for performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information to obtain an object region point cloud; A 3D model generation module for obtaining a 3D model of the object by performing reconstruction processing on the object region point cloud; The object region point cloud generation module includes: A mapping relationship acquisition unit for obtaining the mapping relationship of the point clouds in the first sparse point cloud information and the second sparse point cloud information, where the mapping relationship is the mapping connection between the 2D object point cloud and the 3D object point cloud; A point cloud screening unit for performing point cloud screening on the first sparse point cloud information and the second sparse point cloud information based on the mapping relationship to extract the point cloud of the object region in the target image and obtain the object region point cloud.

7. The three-dimensional model generation device based on an image sequence according to claim 6, characterized in that The first sparse point cloud information acquisition module includes: An image feature extraction unit for, for each of the target images, obtaining the image features of each of the target images by performing feature extraction of scale-invariant feature transform on the target images; An object sparse point cloud information acquisition unit for performing feature matching on the image features of multiple target images to obtain the object sparse point cloud information; An object sparse point cloud information reconstruction unit for performing reconstruction processing on the object sparse point cloud information using an incremental structure from motion algorithm to obtain the first sparse point cloud information.

8. A computer device, characterized in that, It includes a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the 3D model generation method based on an image sequence as described in any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the 3D model generation method based on an image sequence as described in any one of claims 1 to 5 is implemented.

Images