Image-based three-dimensional surface shape reconstruction method and device, equipment and medium

Abstract

The embodiment of the invention provides an image-based three-dimensional surface shape reconstruction method and device, equipment and a medium, relates to the technical field of three-dimensional measurement, and is suitable for the field of financial science and technology and the field of digital medical treatment. The method comprises the following steps: segmenting an interference fringe pattern into sub-fringe images for image quality evaluation; determining an image preprocessing mode according to the image quality of the sub-stripes, and performing image preprocessing on the sub-stripe images according to the determined image preprocessing mode; extracting phase information of the sub-stripe images and then splicing the sub-stripe images into a phase image; and performing three-dimensional surface shape reconstruction according to the phase image. According to the embodiment of the invention, the method can improve the quality of the sub-stripe image in a targeted manner, further extracts the local phase information of the sub-stripe image through the neural network, improves the precision of three-dimensional modeling, and is especially suitable for the three-dimensional surface shape reconstruction of a complex surface.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional measurement technology, and is applicable to the fields of financial technology and digital medicine, and in particular to an image-based three-dimensional surface reconstruction method and apparatus, equipment and medium. Background Art

[0002] Three-dimensional measurement technology is a technology that performs all-round measurement of objects. Three-dimensional measurement includes active three-dimensional measurement and passive three-dimensional measurement. Active three-dimensional measurement is used to obtain deformed fringe images by adding structured light sources, and calculate the three-dimensional surface shape of the measured object based on the deformed fringe images. Active three-dimensional measurement has high measurement accuracy and can be applied to technologies such as three-dimensional face acquisition and three-dimensional tooth modeling. For example, in financial technology scenarios, the three-dimensional surface contour of the face can be obtained for face recognition to confirm the user's identity; in digital medical scenarios, the three-dimensional data of the patient's part to be diagnosed can be obtained to customize medical devices for the patient. However, active three-dimensional measurement is affected by many factors, and there are measurement errors, which affect the accuracy of the three-dimensional surface shape. Therefore, how to improve the accuracy of the three-dimensional surface shape of objects has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an image-based three-dimensional surface reconstruction method and device, equipment and medium, aiming to improve the three-dimensional surface accuracy.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an image-based three-dimensional surface reconstruction method, the method comprising:

[0005] Acquire a target deformed stripe image of a target object, and segment the target deformed stripe image to obtain sub-stripe images;

[0006] Performing image quality assessment on the sub-stripe image to obtain sub-stripe image quality data;

[0007] Determining an image preprocessing method for the sub-stripe image according to the sub-stripe image quality data;

[0008] According to the image preprocessing method, the sub-stripe image is preprocessed to obtain a target sub-stripe image;

[0009] Extracting phase information of the target sub-fringe image through a pre-trained neural network to obtain a sub-phase image;

[0010] Perform image stitching based on the sub-phase images to obtain a target phase image;

[0011] The three-dimensional surface shape of the target object is reconstructed according to the target phase image.

[0012] In some embodiments, reconstructing the three-dimensional shape of the target object according to the target phase image includes:

[0013] Performing phase unwrapping on the target phase image to obtain an absolute phase;

[0014] Performing Fourier filtering on the absolute phase to obtain pure phase information;

[0015] The three-dimensional coordinates of the target object are calculated according to the pure phase information to obtain the three-dimensional surface shape of the target.

[0016] In some embodiments, performing image stitching based on the sub-phase images to obtain a target phase image includes:

[0017] Extracting feature points from the sub-phase image to obtain contour feature points;

[0018] Based on the contour feature points, a phase image pair is screened out from the sub-phase images; wherein the phase image pair includes two sub-phase images, and the two sub-phase images in the phase image pair have phase information of the same contour point of the target object;

[0019] transforming the phase image pair to a target coordinate system;

[0020] Based on the target coordinate system, stitching the phase image pair to obtain a stitched image;

[0021] The stitched images are fused to obtain a target phase image.

[0022] In some embodiments, acquiring a target deformed stripe image of a target object includes:

[0023] Acquire an original deformed fringe image of the target object; wherein the original deformed fringe image is an image captured based on a preset shooting angle;

[0024] Performing image quality evaluation on the original deformed fringe image to obtain deformed fringe image quality data;

[0025] The target deformed fringe image is screened out from the deformed fringe images according to the deformed fringe image quality data.

[0026] In some embodiments, the step of selecting the target deformed fringe image from the deformed fringe images according to the deformed fringe image quality data comprises:

[0027] Filtering candidate deformed fringe images from the deformed fringe images according to the deformed fringe image quality data;

[0028] Image stitching is performed based on the candidate deformed stripe images to obtain the target deformed stripe image.

[0029] In some embodiments, performing image quality assessment on the sub-stripe image to obtain sub-stripe image quality data includes:

[0030] Calculating the light intensity of stripes in the sub-stripe image to obtain the stripe light intensity;

[0031] Based on the stripe light intensity, calculating the modulation degree of the sub-stripe image to obtain the image modulation degree;

[0032] Performing wavelet transform on the sub-stripe image to obtain wavelet coefficients;

[0033] Based on the wavelet coefficients, the clarity of the sub-stripe image is calculated to obtain image clarity;

[0034] The sub-stripe image quality data is obtained by performing weighted summation according to the image modulation degree and the image clarity.

[0035] In some embodiments, determining the image preprocessing method of the sub-stripe image according to the sub-stripe image quality data includes:

[0036] Obtain reference quality data;

[0037] Performing a quality comparison between the sub-stripe image quality data and the reference quality data to obtain image quality difference data;

[0038] An image restoration method or an image noise reduction method is used as the image preprocessing method according to the poor image quality data.

[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a device for reconstructing a three-dimensional surface shape of an object, the device comprising:

[0040] An image segmentation module is used to obtain a target deformed stripe image of a target object, and segment the target deformed stripe image to obtain sub-stripe images;

[0041] An image quality assessment module, used to perform image quality assessment on the sub-stripe image to obtain sub-stripe image quality data;

[0042] An image preprocessing method determining module, used for determining an image preprocessing method of the sub-stripe image according to the sub-stripe image quality data;

[0043] An image preprocessing module, used to perform image preprocessing on the sub-stripe image according to the image preprocessing method to obtain a target sub-stripe image;

[0044] A phase extraction module, used to extract the phase information of the target sub-fringe image through a pre-trained neural network to obtain a sub-phase image;

[0045] An image stitching module, used for performing image stitching based on the sub-phase images to obtain a target phase image;

[0046] A surface reconstruction module is used to reconstruct the three-dimensional surface of the target object according to the target phase image.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The present application proposes an image-based three-dimensional surface reconstruction method and device, equipment and medium, which divide a target deformed stripe image into sub-stripe images to perform local processing on the target deformed stripe image; perform quality assessment on the sub-stripe images, and determine the image preprocessing method of the sub-stripe images based on the image quality assessment data, and improve the quality of the sub-stripe images in a targeted manner through quality assessment-driven preprocessing; extract phase information from the preprocessed sub-stripe images through a neural network to obtain sub-phase images, then splice the sub-phase images to obtain a target phase image, establish a three-dimensional surface based on the target phase image, further extract local phase information, and improve the three-dimensional modeling accuracy, which is particularly suitable for three-dimensional surface reconstruction of complex surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of an image-based three-dimensional surface reconstruction method provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0052] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0053] Figure 4 yes Figure 1 Flow chart of step S102 in FIG.

[0054] Figure 5 yes Figure 1 Flow chart of step S103 in FIG.

[0055] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0056] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0057] Figure 8 is a schematic diagram of the structure of a device for reconstructing the three-dimensional surface shape of an object provided in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Active 3D Measurement (A3M): A technology that projects a light source onto the object to be measured, and uses a camera to capture the surface image of the object to be measured that includes the reflected light or transmitted light from the light source, and uses a mathematical model established by the imaging geometry to solve the 3D structural information of the object to be measured. The advantages of active 3D measurement are its non-contact nature, high automation, high sensitivity, and high precision. Active 3D measurement technology is widely used in industrial inspection, reverse engineering, biomedicine, machine vision and other fields. For example, it is used for 3D face acquisition, 3D tooth modeling, and 3D information recording and profiling of various physical models.

[0064] Image Quality Assessment (IQA): Image quality is an important indicator for comparing the performance of various image processing methods and optimizing system parameters. Therefore, it is of great significance to establish an effective image quality assessment mechanism in the fields of image acquisition, coding compression, network transmission, etc. Image quality assessment methods are divided into subjective evaluation methods and objective evaluation methods. The subjective evaluation method is based on the individual's subjective evaluation of the image; the objective evaluation method sets evaluation indicators based on the basic principles of digital image processing to measure the quality of image processing methods.

[0065] Image Preprocessing (IP): Used to enhance image quality before further analysis or processing. Image preprocessing improves the accuracy of subsequent image processing methods by reducing image noise, correcting distortion, and image inversion.

[0066] Neural Network (NN): A computational model that mimics the way neurons in the human brain connect and interact. It consists of a large number of nodes, which are usually organized into layers. Each node can receive input signals, perform weighted summation, and generate outputs through a nonlinear activation function. Neural networks learn patterns and features in data by adjusting the connection weights between neurons.

[0067] Image Stitching (IS): is a technique that combines multiple overlapping images into a larger image, which is widely used in medical imaging, computer vision, satellite data, and automatic recognition of military targets. It involves steps such as feature extraction, image registration, image deformation, and image fusion to achieve smooth transition and seamless stitching.

[0068] Phase Unwrapping (PU): The process of calculating the truncated phase based on the fringe image is called phase extraction. The truncated phase appears as a discontinuous sawtooth shape and cannot truly reflect the phase information of the surface morphology of the object. In order to obtain the continuous surface morphology of the object, it is necessary to perform phase unwrapping on the basis of the truncated phase to obtain continuous phase information.

[0069] Phase-Only Information (PI): refers to information that contains phase changes but ignores amplitude changes.

[0070] In the related art, three-dimensional modeling is performed on objects with complex surfaces through active three-dimensional measurement. However, due to shadows or mutations on the object surface, the interference fringe pattern obtained by the imaging system will have broken or misaligned fringes, and the phase information obtained in the shadow area or mutation point is often unreliable, which affects the three-dimensional surface reconstruction of objects with complex surfaces.

[0071] Based on this, the embodiments of the present application provide an image-based three-dimensional surface reconstruction method and apparatus, device and medium, aiming to improve the three-dimensional surface accuracy.

[0072] The image-based three-dimensional surface reconstruction method, device, equipment and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, an image-based three-dimensional surface reconstruction method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The image-based three-dimensional surface reconstruction method provided in the embodiment of the present application relates to the field of three-dimensional measurement technology. The image-based three-dimensional surface reconstruction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements an image-based three-dimensional surface reconstruction method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Figure 1 is an optional flow chart of the image-based three-dimensional surface reconstruction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0079] Step S101, obtaining a target deformed stripe image of a target object, segmenting the target deformed stripe image to obtain sub-stripe images;

[0080] Step S102, performing image quality assessment on the sub-stripe image to obtain sub-stripe image quality data;

[0081] Step S103, determining an image preprocessing method of the sub-stripe image according to the sub-stripe image quality data;

[0082] Step S104, performing image preprocessing on the sub-stripe image according to an image preprocessing method to obtain a target sub-stripe image;

[0083] Step S105, extracting the phase information of the target sub-fringe image through a pre-trained neural network to obtain a sub-phase image;

[0084] Step S106, performing image stitching based on the sub-phase images to obtain a target phase image;

[0085] Step S107: reconstructing the three-dimensional shape of the target object according to the target phase image.

[0086] In steps S101 to S107 shown in the embodiment of the present application, the target deformed stripe image is divided into sub-stripe images to perform local processing on the target deformed stripe image; the quality of the sub-stripe images is evaluated, and the image preprocessing method of the sub-stripe images is determined according to the image quality evaluation data, and the quality of the sub-stripe images is targetedly improved through quality evaluation-driven preprocessing; phase information is extracted from the preprocessed sub-stripe images through a neural network to obtain sub-phase images, and then the sub-phase images are spliced ​​to obtain a target phase image, and a three-dimensional surface shape is established based on the target phase image, and local phase information is further extracted to improve the three-dimensional modeling accuracy, which is particularly suitable for three-dimensional surface reconstruction of complex surfaces.

[0087] In step S101 of some embodiments, the target object may be a human body part, such as a face, limbs, or teeth; or an industrial part, such as a camshaft, pulley, gear, or spline; but not limited thereto. A stripe grating is projected onto the target object, and the stripe grating is deformed by being highly modulated by the contour points of the target object. The deformed stripe grating is photographed by an imaging device to obtain a target deformed stripe image. Specifically, in an insurance claim scenario, the limbs can be reconstructed in three dimensions to obtain the three-dimensional shape of the limbs as evidence for the claim; in a financial business application scenario, the face can be reconstructed in three dimensions to obtain the three-dimensional shape of the face as identity authentication material; in a digital medical scenario, the injured part of the patient can be reconstructed in three dimensions to obtain the three-dimensional shape of the injured part in order to customize a prosthesis.

[0088] In step S101 of some embodiments, a deformed stripe image captured in real time may be acquired by a mobile terminal as a target deformed stripe image, or a deformed stripe image captured in advance may be acquired by a server as a target deformed stripe image, without limitation thereto.

[0089] In step S101 of some embodiments, the target deformed stripe image may be segmented into a plurality of rectangular sub-stripe images; or each stripe in the target deformed stripe image may be identified and segmented to obtain a plurality of sub-stripe images; but the present invention is not limited thereto.

[0090] In some embodiments, before step 102, the image-based three-dimensional surface reconstruction method further includes: calculating the modulation degree of the sub-strip image; adjusting the size of the sub-strip image according to the modulation degree of the sub-strip image. Specifically, the modulation degree of each pixel in the sub-strip image is calculated to obtain the maximum modulation degree, the minimum modulation degree, and the average modulation degree; the average modulation degree of each pixel in the sub-strip image is taken as the modulation degree of the sub-strip image; the difference between the maximum modulation degree and the modulation degree of the sub-strip image is calculated to obtain the first difference; the difference between the maximum modulation degree and the minimum modulation degree is calculated to obtain the second difference; the ratio of the first difference to the second difference is calculated to obtain the image size weight; according to the image size weight, the weighted value of the original size of the sub-strip image is calculated to obtain the size modification value; the sum of the original size of the sub-strip image and the size modification value is calculated as the target size of the sub-strip image.

[0091] In this embodiment, the size of the sub-stripe image is adjusted according to the modulation degree of the sub-stripe image, so that the smaller the modulation degree of the sub-stripe image, the larger the size of the sub-stripe image, so as to increase the stripe details of the sub-stripe image with poor modulation degree and improve the accuracy of the subsequently extracted phase information; at the same time, the area of ​​the sub-stripe image is increased, and the overlapping area between adjacent sub-stripe images is increased, so as to improve the accuracy of subsequent image stitching.

[0092] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0093] Step S201, obtaining an original deformed stripe image of a target object; wherein the original deformed stripe image is an image captured based on a preset shooting angle;

[0094] Step S202, performing image quality assessment on the original deformed fringe image to obtain deformed fringe image quality data;

[0095] Step S203: Filter out target deformed fringe images from the deformed fringe images according to the deformed fringe image quality data.

[0096] In steps S201 to S203 shown in the embodiment of the present application, quality assessment is performed by acquiring original deformed stripe images under multiple shooting angles, and high-quality target deformed stripe images are screened out for three-dimensional surface modeling, thereby improving the accuracy of three-dimensional surface modeling.

[0097] In step S201 of some embodiments, the original deformed stripe images obtained by taking multiple shots at different shooting angles may be acquired by changing the camera position; or the original deformed stripe images obtained by taking multiple shots at different shooting angles may be acquired by rotating the target object; but this is not limited thereto.

[0098] In step S202 of some embodiments, when performing image quality assessment on the original deformed stripe image, the spatial frequency of the stripe image, the maximum phase difference between adjacent pixels in the stripe image, the modulation degree or the clarity may be used as image quality indicators, and the deformed stripe image quality data may be calculated based on at least one image quality indicator, but is not limited thereto.

[0099] In step S203 of some embodiments, the original deformed stripe image with the highest deformed stripe image quality data can be selected as the target deformed stripe image; or a stripe image quality threshold can be set. If the deformed stripe image quality data of the acquired original deformed stripe image is less than the stripe image quality threshold, a new original deformed stripe image is acquired again until an original deformed stripe image with a deformed stripe image quality data greater than or equal to the stripe image quality threshold is obtained, and the original deformed stripe image is used as the target deformed stripe image; it is not limited to this.

[0100] See also Figure 3 In some embodiments, step S203 may include but is not limited to steps S301 to S302:

[0101] Step S301, selecting candidate deformed fringe images from the deformed fringe images according to the deformed fringe image quality data;

[0102] Step S302: performing image stitching based on the candidate deformed stripe images to obtain a target deformed stripe image.

[0103] In steps S301 to S302 shown in the embodiment of the present application, quality assessment is performed by acquiring the original deformed stripe images under multiple shooting angles, high-quality candidate deformed stripe images are screened out for image stitching, and the target deformed stripe images are obtained for three-dimensional surface modeling. The stripe quality is optimized by integrating the stripe information under different angles, thereby further improving the accuracy of three-dimensional surface modeling.

[0104] In step S102 of some embodiments, the sub-stripe image quality data may be a sub-stripe image quality score value calculated based on an image quality index, or may be a sub-stripe image quality level obtained by dividing the sub-stripe image quality score value, but is not limited thereto.

[0105] In step S102 of some embodiments, when performing image quality assessment on a sub-stripe image, the spatial frequency of the stripe image, the maximum phase difference between adjacent pixels in the stripe image, the modulation degree or clarity may be used as image quality indicators to calculate the sub-stripe image quality data, but is not limited thereto.

[0106] See also Figure 4 In some embodiments, step S102 includes but is not limited to steps S401 to S405:

[0107] Step S401, calculating the light intensity of the stripes in the sub-stripe image to obtain the stripe light intensity;

[0108] Step S402, calculating the modulation degree of the sub-fringe image based on the fringe light intensity to obtain the image modulation degree;

[0109] Step S403, performing wavelet transform on the sub-stripe image to obtain wavelet coefficients;

[0110] Step S404, calculating the definition of the sub-stripe image based on the wavelet coefficients to obtain the image definition;

[0111] Step S405, performing weighted summation according to the image modulation degree and the image clarity to obtain sub-stripe image quality data.

[0112] In steps S401 to S405 shown in the embodiment of the present application, the modulation degree of the sub-stripe image is calculated based on the stripe light intensity; the clarity of the sub-stripe image is calculated based on the wavelet coefficients; and the sub-stripe image quality data is obtained by combining the image modulation degree and the image clarity, so as to facilitate the subsequent determination of the image preprocessing method of the stripe image based on the image quality, which helps to improve the accuracy of subsequent phase information extraction and phase unfolding, thereby improving the accuracy of three-dimensional surface modeling.

[0113] In step S403 of some embodiments, a non-subsampled contourlet transform may be performed on the sub-stripe image, or a two-dimensional discrete wavelet transform may be performed on the sub-stripe image, but the present invention is not limited thereto.

[0114] In some embodiments, the wavelet coefficients include sub-band coefficients at multiple scales and directions, and step S404 includes: determining the weight of each sub-band coefficient; and based on the weight of each sub-band coefficient, calculating a weighted average of the sub-band coefficients at multiple scales and directions to obtain image clarity.

[0115] See also Figure 5 In some embodiments, step S103 may include but is not limited to steps S501 to S503:

[0116] Step S501, obtaining reference quality data;

[0117] Step S502, performing a quality comparison between the sub-strip image quality data and the reference quality data to obtain image quality difference data;

[0118] Step S503: using an image restoration method or an image noise reduction method as an image preprocessing method according to the image quality difference data.

[0119] In steps S501 to S503 shown in the embodiment of the present application, based on the quality of the sub-stripe image, image restoration is selected for low-quality sub-stripe images to improve damaged or missing parts in the sub-stripe image, such as broken or misaligned stripes; and image noise reduction is performed on high-quality sub-stripe images to remove noise while maintaining stripe details as much as possible.

[0120] In step S501 of some embodiments, a quality threshold may be set as the reference quality data, or a quality interval may be set as the reference quality data, but the present invention is not limited thereto.

[0121] It is easy to understand that in step S502 of some embodiments, when the sub-stripe image quality data is a sub-stripe image quality score, if the sub-stripe image quality score is greater than or equal to the quality threshold, image denoising is performed on the sub-stripe image; if the sub-stripe image quality score is less than the quality threshold, image restoration is performed on the sub-stripe image.

[0122] In step S103 of some embodiments, the image preprocessing method may be image filtering, such as mean filtering and median filtering; the image preprocessing method may also be image restoration, such as removing burrs and connecting breakpoints; the image preprocessing method may also be image enhancement, such as super-resolution and contrast enhancement; the image preprocessing method is not limited to this.

[0123] In some embodiments, the image preprocessing method is mean filtering, and step S104 includes: defining a sliding window; traversing each pixel point of the sub-stripe image, calculating the average pixel value of the pixel points in the sliding window for the currently visited pixel point, and using the average pixel value to update the pixel value of the currently visited pixel point.

[0124] In some embodiments, the image preprocessing method is to connect breakpoints, and step S104 includes: identifying the endpoints of the broken stripes in the sub-stripe image; calculating the tangent slope of the endpoints and the distance between the endpoints; detecting matching endpoint pairs based on the tangent slope and the distance; and connecting matching breakpoint pairs.

[0125] Before step S105 in some embodiments, the image-based three-dimensional surface reconstruction method further includes: constructing a neural network model; collecting deformed fringe images, overlapping and blocking each deformed fringe image to obtain a plurality of sub-deformed fringe images. The sub-deformed fringe images are used to train the neural network model, learn the mapping relationship between the fringe image and the phase image, and calculate the loss value between the output value of the neural network model and the true value; based on the loss value, the neural network model parameters are continuously iteratively optimized through the back propagation path until convergence.

[0126] See also Figure 6 In some embodiments, step S106 may include but is not limited to steps S601 to S605:

[0127] Step S601, extracting feature points from the sub-phase image to obtain contour feature points;

[0128] Step S602, based on the contour feature points, a phase image pair is screened out from the sub-phase images; wherein the phase image pair includes two sub-phase images, and the two sub-phase images in the phase image pair have phase information of the same contour point of the target object;

[0129] Step S603, converting the phase image pair to a target coordinate system;

[0130] Step S604, stitching the phase image pair based on the target coordinate system to obtain a stitched image;

[0131] Step S605: performing image fusion on the stitched images to obtain a target phase image.

[0132] Steps S601 to S605 shown in the embodiment of the present application, through feature point extraction and image matching, obtain sub-phase images with phase information of the same contour point of the target object for image stitching and image fusion, which can achieve accurate and robust image stitching to improve the accuracy of subsequent three-dimensional surface modeling.

[0133] In step S601 of some embodiments, feature points of the sub-phase image can be extracted by the SIFT method. The SIFT method calculates the difference between Gaussian blurred images of adjacent scales to construct a differential Gaussian pyramid. In the differential Gaussian pyramid, the local extreme values ​​of each pixel point in space and scale are detected, and the extreme value points are used as contour feature points. Feature points of the sub-phase image can also be extracted by the Harris corner point detection method. The Harris corner point detection method detects the severity of the grayscale value change in the image to obtain contour feature points. It is not limited to this.

[0134] In step S602 of some embodiments, the phase image pair may be obtained by a K-nearest neighbor matching method; or the phase image pair may be obtained by a random sampling consensus method (RANSAC), but is not limited thereto.

[0135] In step S605 of some embodiments, image fusion can be performed using a weighted average method, which performs weighted addition of pixel values ​​in the overlapping areas of the two images to obtain pixel values ​​of the spliced ​​image, so as to eliminate image splicing traces and band-like areas; image fusion can also be performed using a Laplacian pyramid, where the overlapping areas of the two images are decomposed into different frequency bands, and pixel values ​​in the overlapping areas are weighted added to obtain pixel values ​​of the spliced ​​image, so as to eliminate splicing seams and ghost images; this is not limited to this.

[0136] See also Figure 7In some embodiments, step S107 may include but is not limited to steps S701 to S703:

[0137] Step S701, performing phase unwrapping on the target phase image to obtain an absolute phase;

[0138] Step S702, performing Fourier filtering on the absolute phase to obtain pure phase information;

[0139] Step S703, calculating the three-dimensional coordinates of the target object according to the pure phase information to obtain the target three-dimensional surface shape.

[0140] In steps S701 to S703 shown in the embodiment of the present application, phase unfolding is performed on the target phase image to obtain the absolute phase, so as to obtain the exact phase value of each pixel; Fourier filtering is used to remove noise and unnecessary frequency components to obtain pure phase information; the relationship between the phase information and the surface height of the object is used to calculate the three-dimensional coordinates of the target object based on the pure phase information to reconstruct the three-dimensional surface shape of the object surface, which has high measurement accuracy and is suitable for three-dimensional surface shape reconstruction of complex surfaces.

[0141] In step S701 of some embodiments, the target phase image can be phase-unwrapped by a spatial phase unwrapping method, which unwraps the phase based on the phase difference between adjacent points; the target phase image can also be phase-unwrapped by a temporal phase unwrapping method, which uses fringe phase information of multiple frequencies to perform step-by-step phase unwrapping, and has good adaptability to the measurement of static targets, such as the negative exponential fitting method, the three-frequency heterodyne method, the three-frequency method, and the dual-frequency method; or the target phase image can be phase-unwrapped by a phase unwrapping method based on deep learning; but it is not limited to this.

[0142] In the financial technology scenario, taking the human face as the target object, taking user credit identity verification as an example, the image-based 3D face reconstruction method includes the following steps:

[0143] The projector projects a stripe grating on the customer's face, and the camera collects the deformed stripe grating highly modulated by the face surface to obtain the target deformed stripe image;

[0144] According to the preset sub-strip image size and shape, the target deformed stripe image is segmented to obtain sub-strip images; in this embodiment, the target deformed stripe image is segmented into a plurality of rectangular sub-strip images;

[0145] Using modulation and definition as image quality evaluation indicators, the image quality of the sub-stripe image is evaluated to obtain the sub-stripe image quality data;

[0146] Determining an image preprocessing method for the sub-stripe image according to the sub-stripe image quality data;

[0147] According to the image preprocessing method, the sub-stripe image is preprocessed to obtain a target sub-stripe image; in this embodiment, when the sub-stripe image quality data is less than a preset threshold, the sub-stripe image is breakpoint connected to obtain a target sub-stripe image; when the sub-stripe image quality data is greater than or equal to the preset threshold, the sub-stripe image is mean filtered to obtain a target sub-stripe image;

[0148] The phase information of the target sub-fringe image is extracted through a pre-trained neural network to obtain a sub-phase image;

[0149] Perform image stitching based on the sub-phase images to obtain a target phase image;

[0150] Reconstruct the three-dimensional face shape of the customer's face according to the target phase image to obtain the three-dimensional face shape of the target face;

[0151] Obtaining coordinates of two-dimensional facial feature points, mapping the coordinates of the two-dimensional facial feature points to the three-dimensional face shape of the target face, and obtaining three-dimensional facial feature points;

[0152] Based on the three-dimensional facial feature points, the customer's identity is authenticated and the identity authentication result is output; if the authentication result is successful, the user's credit application is approved; otherwise, the user's credit application is not approved, and other security measures are taken to confirm the user's identity.

[0153] In the digital medical scenario, the patient's oral cavity is taken as the target object, and the three-dimensional face shape construction of the patient's oral cavity is taken as an example. A three-dimensional face shape reconstruction method based on an image includes the following steps:

[0154] The three-dimensional measurement endoscope projects a fringe grating into the patient's oral cavity and collects a deformed fringe image highly modulated by the oral cavity surface to obtain a target deformed fringe image;

[0155] Identify and segment each stripe in the target deformed stripe image to obtain a sub-stripe image;

[0156] The maximum phase difference between adjacent pixels in the sub-stripe image is calculated as an image quality evaluation index; based on the image quality evaluation index, the image quality of the sub-stripe image is evaluated to obtain the sub-stripe image quality data;

[0157] According to the sub-stripe image quality data, an image preprocessing method of the sub-stripe image is determined; in this embodiment, when the sub-stripe image quality data is less than a preset threshold, the sub-stripe image is deburred to obtain a target sub-stripe image; when the sub-stripe image quality data is greater than or equal to the preset threshold, the sub-stripe image is median filtered to obtain a target sub-stripe image;

[0158] According to the image preprocessing method, the sub-stripe image is preprocessed to obtain the target sub-stripe image;

[0159] The phase information of the target sub-fringe image is extracted through a pre-trained neural network to obtain a sub-phase image;

[0160] Perform image stitching based on the sub-phase images to obtain a target phase image;

[0161] The three-dimensional shape of the oral cavity is reconstructed according to the target phase image to obtain the target three-dimensional shape of the oral cavity.

[0162] See also Figure 8 The embodiment of the present application further provides a device for reconstructing the three-dimensional surface shape of an object, which can implement the above-mentioned image-based three-dimensional surface shape reconstruction method, and the device includes:

[0163] An image segmentation module is used to obtain a target deformed stripe image of a target object, and segment the target deformed stripe image to obtain a sub-stripe image;

[0164] An image quality assessment module is used to perform image quality assessment on the sub-stripe image to obtain sub-stripe image quality data;

[0165] An image preprocessing method determination module is used to determine an image preprocessing method of a sub-stripe image according to the sub-stripe image quality data;

[0166] An image preprocessing module is used to perform image preprocessing on the sub-stripe image according to an image preprocessing method to obtain a target sub-stripe image;

[0167] A phase extraction module is used to extract the phase information of the target sub-fringe image through a pre-trained neural network to obtain a sub-phase image;

[0168] An image stitching module, used for performing image stitching based on the sub-phase images to obtain a target phase image;

[0169] The surface reconstruction module is used to reconstruct the three-dimensional surface of the target object according to the target phase image.

[0170] The specific implementation of the object three-dimensional surface reconstruction device is basically the same as the specific embodiment of the above-mentioned image-based three-dimensional surface reconstruction method, and will not be repeated here.

[0171] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image-based three-dimensional surface reconstruction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0172] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0173] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0174] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes a method for image-based three-dimensional surface reconstruction in the embodiment of the present application;

[0175] Input / output interface 903, used to implement information input and output;

[0176] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0177] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0178] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0179] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image-based three-dimensional face reconstruction method is implemented.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The embodiments of the present application provide an image-based three-dimensional surface reconstruction method, an object three-dimensional surface reconstruction device, an electronic device and a storage medium, which divide a target deformed stripe image into sub-stripe images to perform local processing on the target deformed stripe image; perform quality assessment on the sub-stripe images, and determine an image preprocessing method for the sub-stripe images based on image quality assessment data, and improve the quality of the sub-stripe images in a targeted manner through quality assessment-driven preprocessing; extract phase information from the preprocessed sub-stripe images through a neural network to obtain sub-phase images, and then splice the sub-phase images to obtain a target phase image, establish a three-dimensional surface based on the target phase image, further extract local phase information, and improve the three-dimensional modeling accuracy, which is particularly suitable for three-dimensional surface reconstruction of complex surfaces.

[0182] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0183] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0186] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0187] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0192] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for image-based three-dimensional surface reconstruction, characterized in that: The method comprises: Acquire a target deformed stripe image of a target object, and segment the target deformed stripe image to obtain sub-stripe images; Performing image quality assessment on the sub-stripe image to obtain sub-stripe image quality data; Determining an image preprocessing method for the sub-stripe image according to the sub-stripe image quality data; According to the image preprocessing method, the sub-stripe image is preprocessed to obtain a target sub-stripe image; Extracting phase information of the target sub-fringe image through a pre-trained neural network to obtain a sub-phase image; Perform image stitching based on the sub-phase images to obtain a target phase image; The three-dimensional surface shape of the target object is reconstructed according to the target phase image.

2. The method according to claim 1, characterized in that The three-dimensional surface reconstruction of the target object according to the target phase image comprises: Performing phase unwrapping on the target phase image to obtain an absolute phase; Performing Fourier filtering on the absolute phase to obtain pure phase information; The three-dimensional coordinates of the target object are calculated according to the pure phase information to obtain the three-dimensional surface shape of the target.

3. The method according to claim 1, characterized in that The step of performing image stitching based on the sub-phase images to obtain a target phase image includes: Extracting feature points from the sub-phase image to obtain contour feature points; Based on the contour feature points, a phase image pair is screened out from the sub-phase images; wherein the phase image pair includes two sub-phase images, and the two sub-phase images in the phase image pair have phase information of the same contour point of the target object; transforming the phase image pair to a target coordinate system; Based on the target coordinate system, stitching the phase image pair to obtain a stitched image; The stitched images are fused to obtain a target phase image.

4. The method according to claim 1, characterized in that: The step of acquiring a target deformed stripe image of a target object comprises: Acquire an original deformed fringe image of the target object; wherein the original deformed fringe image is an image captured based on a preset shooting angle; Performing image quality evaluation on the original deformed fringe image to obtain deformed fringe image quality data; The target deformed fringe image is screened out from the deformed fringe images according to the deformed fringe image quality data.

5. The method according to claim 4, characterized in that The step of selecting the target deformed fringe image from the deformed fringe image according to the deformed fringe image quality data comprises: Filtering candidate deformed fringe images from the deformed fringe images according to the deformed fringe image quality data; Image stitching is performed based on the candidate deformed stripe images to obtain the target deformed stripe image.

6. The method according to any one of claims 1 to 5, characterized in that: The performing image quality assessment on the sub-stripe image to obtain sub-stripe image quality data includes: Calculating the light intensity of stripes in the sub-stripe image to obtain the stripe light intensity; Based on the stripe light intensity, calculating the modulation degree of the sub-stripe image to obtain the image modulation degree; Performing wavelet transform on the sub-stripe image to obtain wavelet coefficients; Based on the wavelet coefficients, the clarity of the sub-stripe image is calculated to obtain image clarity; The sub-stripe image quality data is obtained by performing weighted summation according to the image modulation degree and the image clarity.

7. The method according to claim 6, characterized in that Determining the image preprocessing method of the sub-stripe image according to the sub-stripe image quality data includes: Obtain reference quality data; Performing a quality comparison between the sub-stripe image quality data and the reference quality data to obtain image quality difference data; An image restoration method or an image noise reduction method is used as the image preprocessing method according to the poor image quality data.

8. A device for reconstructing a three-dimensional surface shape of an object, characterized in that: The device comprises: An image segmentation module is used to obtain a target deformed stripe image of a target object, and segment the target deformed stripe image to obtain sub-stripe images; An image quality assessment module, used to perform image quality assessment on the sub-stripe image to obtain sub-stripe image quality data; An image preprocessing method determining module, used for determining an image preprocessing method of the sub-stripe image according to the sub-stripe image quality data; An image preprocessing module, used to perform image preprocessing on the sub-stripe image according to the image preprocessing method to obtain a target sub-stripe image; A phase extraction module, used to extract the phase information of the target sub-fringe image through a pre-trained neural network to obtain a sub-phase image; An image stitching module, used for performing image stitching based on the sub-phase images to obtain a target phase image; A surface reconstruction module is used to reconstruct the three-dimensional surface of the target object according to the target phase image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image-based three-dimensional surface reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image-based three-dimensional surface reconstruction method according to any one of claims 1 to 7 is implemented.

Images