Three-dimensional reconstruction method and device and storage medium

Abstract

The invention provides a three-dimensional reconstruction method and device and a storage medium, and the method comprises the steps: obtaining a plurality of frames of first images and a plurality of frames of second images, aiming at each first camera pixel in a plurality of first camera pixels of a first camera, according to the plurality of frames of first images, in a plurality of projection pixels, obtaining a plurality of projection pixels; determining a first number of first projection pixels corresponding to the first camera pixels; in a plurality of second camera pixels of a second camera, determining a first number of second camera pixels corresponding to the first number of first projection pixels, the first number of first projection pixels being in one-to-one correspondence with the first number of second camera pixels; in the first number of second camera pixels, determining second camera pixels matched with the first camera pixels; and performing three-dimensional reconstruction on the target object according to the matching relationship between the plurality of first camera pixels and the plurality of second camera pixels, thereby improving the precision of three-dimensional reconstruction.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, device, and storage medium. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, 3D reconstruction technology has been widely used in many fields. Especially in the manufacturing industry, tasks such as 3D detection, precise positioning and surface analysis are becoming increasingly important.

[0003] Currently, 3D reconstruction methods rely on technologies such as laser scanning, stereo vision, or structured light. However, these methods suffer from low 3D reconstruction accuracy when faced with objects with complex reflective properties, especially metal welded workpieces or multi-reflective objects. Summary of the Invention

[0004] Various aspects of the present disclosure provide a three-dimensional reconstruction method, device, and storage medium to improve the accuracy of three-dimensional reconstruction.

[0005] A first aspect of an embodiment of the present disclosure provides a three-dimensional reconstruction method, comprising: acquiring multiple frames of first images and multiple frames of second images, wherein the multiple frames of first images are obtained by capturing the target object with a first camera when a projection machine projects multiple frames of projection images onto the target object, and the multiple frames of second images are obtained by capturing the target object with a second camera when the projection machine projects multiple frames of projection images onto the target object, wherein each frame of the multiple frames of projection images comprises: multiple projection pixels; for each first camera pixel among the multiple first camera pixels of the first camera, determining, from the multiple projection pixels included in the multiple frames of projection images, a first number of first projection pixels corresponding to the first camera pixel; determining, from the multiple second camera pixels of the second camera, based on a correspondence between the multiple frames of second images and the multiple frames of projection images, a first number of second camera pixels corresponding to the first number of first projection pixels, wherein the first number of first projection pixels corresponds one-to-one to the first number of second camera pixels; determining, from the first number of second camera pixels, second camera pixels matching the first camera pixels; and performing three-dimensional reconstruction of the target object based on the matching relationship between the multiple first camera pixels and the multiple second camera pixels.

[0006] A second aspect of the present disclosure provides a three-dimensional reconstruction device, including:

[0007] an acquisition module, configured to acquire a plurality of first image frames and a plurality of second image frames, wherein the plurality of first image frames are acquired by a first camera capturing a target object when the projection optical machine projects the plurality of projection images onto the target object, and the plurality of second image frames are acquired by a second camera capturing the target object when the projection optical machine projects the plurality of projection images onto the target object, wherein each projection image frame in the plurality of projection image frames includes: a plurality of projection pixels;

[0008] a first determining module configured to determine, for each first camera pixel among a plurality of first camera pixels of the first camera, a first number of first projection pixels corresponding to the first camera pixel from a plurality of projection pixels included in the plurality of projection images based on the plurality of first images;

[0009] a second determining module, configured to determine, from a plurality of second camera pixels of the second camera, a first number of second camera pixels corresponding to a first number of first projection pixels based on a correspondence between the plurality of frames of second images and the plurality of frames of projection images, the first number of first projection pixels corresponding to the first number of second camera pixels in a one-to-one correspondence;

[0010] a third determining module, configured to determine, from the first number of second camera pixels, second camera pixels that match the first camera pixels;

[0011] The reconstruction module is used to perform three-dimensional reconstruction of the target object according to the matching relationship between the plurality of first camera pixels and the plurality of second camera pixels.

[0012] A third aspect of an embodiment of the present disclosure provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the three-dimensional reconstruction method of the first aspect when executing the computer program.

[0013] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the three-dimensional reconstruction method of the first aspect.

[0014] A fifth aspect of an embodiment of the present disclosure provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the three-dimensional reconstruction method of the first aspect.

[0015] The present disclosure provides a three-dimensional reconstruction method, comprising: acquiring multiple frames of first images and multiple frames of second images, wherein the multiple frames of first images are acquired by capturing a target object with a first camera when a projection machine projects the multiple frames of projection images onto the target object; and the multiple frames of second images are acquired by capturing the target object with a second camera when the projection machine projects the multiple frames of projection images onto the target object. Each frame of the multiple frames of projection images includes: multiple projection pixels; for each first camera pixel among the multiple first camera pixels of the first camera, determining, from the multiple projection pixels included in the multiple frames of projection images, a first number of first projection pixels corresponding to the first camera pixel; determining, from the multiple second camera pixels of the second camera, based on a correspondence between the multiple frames of second images and the multiple frames of projection images, a first number of second camera pixels corresponding to the first number of first projection pixels, wherein the first number of first projection pixels corresponds one-to-one to the first number of second camera pixels; and determining, from the first number of second camera pixels, second camera pixels matching the first camera pixels; and performing three-dimensional reconstruction of the target object based on the matching relationship between the multiple first camera pixels and the multiple second camera pixels, thereby improving the accuracy of the three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0017] Figure 1 A schematic diagram of an imaging system provided for an exemplary embodiment of the present disclosure;

[0018] Figure 2 A flowchart of a three-dimensional reconstruction method provided by an exemplary embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram of a projected image provided by an exemplary embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram of a first image provided by an exemplary embodiment of the present disclosure Figure 3 ;

[0021] Figure 5 A flowchart of another three-dimensional reconstruction method provided by an exemplary embodiment of the present disclosure;

[0022] Figure 6 A schematic diagram of grayscale values ​​of a first camera pixel in different first images provided by an exemplary embodiment of the present disclosure;

[0023] Figure 7 A structural block diagram of a three-dimensional reconstruction device provided by an exemplary embodiment of the present disclosure;

[0024] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0026] When reconstructing an object in 3D, a projector projects light onto it. Once the light hits the object, a camera captures an image of the object. Then, algorithms are used to generate a point cloud of the object's surface for 3D reconstruction. However, for objects with smooth, highly reflective surfaces and irregular structures like welds or flaws, such as metal welded workpieces, when light is projected onto the object, it reflects the light, which may then be projected back onto the object. If this reflected light is captured by the image, the resulting 3D point cloud may contain missing points, flying points, and other undesirable features, which can affect the accuracy of the object's 3D reconstruction.

[0027] To address the above-mentioned issues, the present disclosure provides a 3D reconstruction method that can determine a first number of first projection pixels corresponding to each first camera pixel, then determine a first number of second camera pixels based on the first number of first projection pixels, and then determine the second camera pixels that match the first camera pixel within the first number of second camera pixels. This improves the accuracy of determining the matching relationship between the first camera pixels and the second camera pixels, thereby improving the 3D reconstruction accuracy of the target object. The present disclosure can avoid the problem of inaccurate matching relationship determination caused by directly determining the matching relationship between the first camera pixels and the second camera pixels.

[0028] Reference Figure 1The present disclosure provides an imaging system 10, which includes: a projector 11, a first camera 12, and a second camera 13. The projector 11 projects light, such as light a1 and light a2, to the target object, and the first camera 12 and the second camera 13 capture images of the target object. For example, light a1 is projected to point Q1 of the target object, and after reflection, light a1 obtains light d1 to the first camera 12 and light d2 to the second camera 13. Light a2 is projected to point Q2 of the target object, and after reflection, light a2 obtains light b1 to the first camera 12 and light b2 to the second camera 13, and light a2 is reflected to obtain reflected light to point Q1. After the light to point Q1 is reflected again, light c1 to the first camera 12 and light c2 to the second camera 13 are obtained. In Figure 1 In the example, the light projected by the projector that is reflected and reaches the first camera 12 and the second camera 13 is a single-reflected light, such as light d1, light b1, light d2, and light b2. The light projected by the projector that is reflected and reaches the first camera 12 and the second camera 13 after at least one more reflection is a multi-reflected light, such as light c1 and light c2. The disclosed embodiments can prevent the impact of multi-reflected light on 3D reconstruction.

[0029] Reference Figure 2 , is a flowchart of a three-dimensional reconstruction method provided by an exemplary embodiment of the present disclosure, the three-dimensional reconstruction method specifically comprising the following steps:

[0030] S201 , acquiring multiple frames of first images and multiple frames of second images.

[0031] Among them, the multiple frames of first images are obtained by the first camera capturing the target object when the projection light machine projects multiple frames of projection images onto the target object, and the multiple frames of second images are obtained by the second camera capturing the target object when the projection light machine projects multiple frames of projection images onto the target object. Each frame of the multiple frames of projection images includes: multiple projection pixels.

[0032] The multiple frames of first images correspond to the multiple frames of projection images in a one-to-one manner, and the multiple frames of second images correspond to the multiple frames of projection images in a one-to-one manner.

[0033] In the embodiment of the present disclosure, the multiple frames of first images and the multiple frames of second images are epipolar rectified.

[0034] It can be understood that controlling the projection light machine to project a frame of projection image corresponds to controlling the first camera to capture a frame of the first image and controlling the second camera to capture a frame of the second image. Figure 3, the multiple frames of projection images are projection image 1, projection image 2, projection image 3, projection image 4, and projection image 5. When the projection optical machine projects a frame of projection image i onto the target object, the first camera captures the first image i of the target image, and the second camera captures the second image i of the target image, where i ranges from 1 to 5.

[0035] In some embodiments, referring to Figure 3 , multiple frames of projection images are projected onto the target object according to a preset timing, and each frame of projection images in the multiple frames of projection images includes: multiple optomechanical pixels, and the multiple optomechanical pixels include multiple projection pixels and multiple non-projection pixels; wherein, the coordinates of the multiple projection pixels included in different frames of projection images in the multiple frames of projection images are different in the multiple optomechanical pixels, and the sum of the multiple projection pixels included in each frame of projection images in the multiple frames of projection images is multiple optomechanical pixels.

[0036] The preset timing refers to projecting multiple frames of projection images in sequence according to the time sequence, for example, Figure 3 , projection image 1, projection image 2, projection image 3, projection image 4 and projection image 5 are projected in chronological order.

[0037] The number N of the plurality of optical-mechanical pixels included in each frame of projection image is the same. For example, referring to Figure 3 , each frame of projection image includes 30 optomechanical pixels, that is, N=30.

[0038] In the embodiment of the present disclosure, the multiple opto-mechanical pixels of each frame of projection image include E projection pixels and F non-projection pixels, where N=E+F.

[0039] It can be understood that a projected pixel is a pixel where the projector projects light, and the projected light illuminates a point on the target object. A non-projected pixel is a pixel where the projector does not project light, and the corresponding position on the target object is dark.

[0040] Furthermore, the multiple optomechanical pixels of each frame of the projected image can be divided into M periods, wherein there is only one projected pixel in each period, that is, N=E×M.

[0041] In the embodiment of the present disclosure, the total projection pixels included in the multiple frames of projection images can cover multiple optical-mechanical pixels of a frame of projection image. Figure 3 , the sum of the projection pixels of projection image 1 to projection image 5 can cover 30 different optical-mechanical pixels.

[0042] In the embodiment of the present disclosure, the horizontal coordinate of the projection light machine corresponding to each projection pixel is as follows: Figure 3 The horizontal coordinate of the projection light machine corresponding to the projection pixel is xj, where j is 1 to 30.

[0043] It can be understood that the present disclosure projects one frame of projection image each time, and each frame of projection image projects multiple rays, each ray corresponds to one projection pixel, and each ray of the next frame of projection image moves one pixel in the horizontal coordinate direction. For example, Figure 3 In the figure, projected image 1 projects 6 rays, and the horizontal coordinates of the corresponding projected pixels are x1, x6, x11, x16, x21, and x26. Projected image 2 also projects 6 rays, and the horizontal coordinates of the corresponding projected pixels are x2, x7, x12, x17, x22, and x27.

[0044] In the embodiments of the present disclosure, other encoding methods may be used, such as Gray code to mark the multiple opto-mechanical pixels of each frame of the projected image, and this is not limited here. Furthermore, the present disclosure may also use other projected images, such as Fourier transform single-pixel projected images, and this is not limited here. Furthermore, the present disclosure does not limit the number of opto-mechanical components included in each frame of the projected image, nor the number of multiple projected images.

[0045] S202 : For each first camera pixel among a plurality of first camera pixels of a first camera, determine a first number of first projection pixels corresponding to the first camera pixel from a plurality of projection pixels included in the plurality of projection images according to the plurality of first image frames.

[0046] In the disclosed embodiments, the multiple first image frames can be understood as being captured at the same location of the target object and at different times. Each first image frame includes multiple pixels, and the number of pixels included in each first image frame is the same. There is a one-to-one correspondence between the multiple pixels included in each first image frame and the multiple first camera pixels.

[0047] exist Figure 4 In the image, the illuminated pixels are the pixels that are illuminated by the light projected by the projector, or the pixels that are illuminated by the reflected light.

[0048] In the embodiment of the present disclosure, the first camera pixel can be represented as P(lx, ly), and the lighting state of each first camera pixel P(lx, ly) in each frame of the first image can be determined based on the projection pixel of the projection light machine corresponding to each frame of the first image.

[0049] For example, if P(l1, l9) is lit in first image 1, the first projected pixel (e.g., projected pixel x1) corresponding to P(l1, l9) in projected image 1 is determined based on this lit state. For example, the light corresponding to projected pixel x1 is directly projected, causing P(l1, l9) to be lit in first image 1. If P(l1, l9) is lit in second image 2, the first projected pixel (e.g., projected pixel x2) corresponding to P(l1, l9) in projected image 2 is determined based on this lit state. For example, the light corresponding to projected pixel x1 is projected onto another point on the target object and reflected, causing P(l1, l9) to be lit in first image 2.

[0050] It can be understood that if the first camera pixel P(lx,ly) is illuminated in the first number of first images, then the first number of first projection pixels can be determined accordingly.

[0051] In the embodiment of the present disclosure, there is no limitation on the method for determining the first camera pixel P(lx,ly) based on the first projection pixel corresponding to the first image according to the lighting state of the first camera pixel P(lx,ly) in the first image.

[0052] S203 : Determine, from a plurality of second camera pixels of the second camera, a first number of second camera pixels corresponding to a first number of first projection pixels according to a correspondence between the plurality of frames of second images and the plurality of frames of projection images.

[0053] The first number of first projection pixels corresponds one-to-one to the first number of second camera pixels.

[0054] In the embodiment of the present disclosure, the number of the plurality of second camera pixels is the same as the number of the plurality of first camera pixels. For example, if the plurality of second camera pixels of the second camera is 400×600, then the plurality of first camera pixels of the first camera is also 400×600.

[0055] Furthermore, there are calibrated conversion relationships between the various structures in the imaging system, such as between the projection optical machine coordinates and the first camera coordinates, between the projection optical machine coordinates and the second camera coordinates, and between the first camera coordinates and the second camera coordinates.

[0056] The first number of second camera pixels may be determined according to the determined first projection pixels and the calibrated conversion relationship.

[0057] For example, for a first camera pixel P(lx, ly), the corresponding multiple first projection pixels can be determined to be first projection pixel t1, first projection pixel t2, ..., first projection pixel tQ, where Q represents a first quantity. Furthermore, the multiple second camera pixels can be determined to be second camera pixel P2(1), second camera pixel P2(2), ..., second camera pixel P2(Q). The first projection pixel tq corresponds to the first projection pixel P2(q), and q ranges from 1 to Q.

[0058] S204 : Determine, from the first number of second camera pixels, second camera pixels that match the first camera pixels.

[0059] In the embodiment of the present disclosure, a cost matching algorithm may be used to determine second camera pixels that match the first camera pixels from the first number of second camera pixels.

[0060] In addition, the second camera pixels matching the first camera pixels may be determined from the first number of second camera pixels in other ways, which are not limited thereto.

[0061] S205 : Perform three-dimensional reconstruction of the target object according to the matching relationship between the plurality of first camera pixels and the plurality of second camera pixels.

[0062] In the disclosed embodiments, the above method can be used to determine the matching relationship between each first camera pixel and each second camera pixel. The matching first camera pixels and second camera pixels are obtained by photographing the same point on the target object. Three-dimensional reconstruction can then be performed based on the matching relationships between the first camera pixels and the second camera pixels.

[0063] In summary, the present disclosure can determine a first number of first projection pixels corresponding to each first camera pixel, then determine a first number of second camera pixels based on the first number of first projection pixels, and determine the second camera pixels that match the first camera pixel from the first number of second camera pixels, thereby improving the accuracy of determining the matching relationship between the first camera pixels and the second camera pixels, thereby improving the three-dimensional reconstruction accuracy of the target object. The present disclosure can avoid the problem of inaccurate matching relationship determination caused by directly determining the matching relationship between the first camera pixels and the second camera pixels.

[0064] Reference Figure 5 , is a flowchart of a three-dimensional reconstruction method provided by an exemplary embodiment of the present disclosure, the three-dimensional reconstruction method specifically comprising the following steps:

[0065] S501: Acquire multiple frames of first images and multiple frames of second images.

[0066] The specific implementation process of this step is referred to S201 and will not be repeated here.

[0067] S502 : Decode the first camera pixel according to the correspondence between the multiple frames of first image and the multiple frames of projection image to obtain multiple first coordinates corresponding to the first camera pixel under the projection light machine.

[0068] Each of the plurality of first coordinates is a coordinate of a corresponding first projection pixel under the projection light engine.

[0069] Reference Figure 6 , represents the grayscale value graph of one of the first camera pixels in different frames of the first image. A grayscale value peak can be understood as the point on the target object corresponding to the first camera pixel being projected onto by light. Only one peak is achieved by a single light reflection; the others are achieved by multiple light reflections.

[0070] Specifically, the timing vector of each first camera pixel P(lx,ly) can be determined based on the multiple frames of the first image, wherein if the dimension of the timing vector of P(lx,ly) is the same as the number of the multiple frames of the first image, the mth dimension of the timing vector indicates whether P(lx,ly) is illuminated in the first image m. If the mth dimension of the timing vector is 1, it indicates that P(lx,ly) is illuminated in the first image m, and if the mth dimension of the timing vector is 0, it indicates that P(lx,ly) is not illuminated in the first image m. For example, referring to Figure 4 For the first camera pixel P(l1, l9), based on multiple frames of the first image, the timing vector of the first camera pixel P(l1, l9) can be determined to be (1, 1, 0, 0, 0). This timing vector indicates that the first camera pixel P(l1, l9) is illuminated in the first image 1 and the second image 2, and is not illuminated in the first image 3 to the first image 5. In this way, the timing vector of each first camera pixel P(lx, ly) can be determined.

[0071] Furthermore, the time series vector is decoded to obtain a plurality of first coordinates corresponding to the first camera pixel under the projection light machine. Figure 3 , the first coordinate is the horizontal coordinate under the projection light machine, and each first coordinate represents a projection pixel.

[0072] In the disclosed embodiment, the plurality of first coordinates corresponding to a first camera pixel indicates that the light at the projection pixel corresponding to the first coordinate causes the point on the target object corresponding to the first camera pixel to be illuminated. The number of the plurality of first coordinates is equal to the number of times the point on the target object corresponding to the first camera pixel is illuminated. For example, for a first camera pixel P(l1, l9), if two first coordinates, such as x1 and x2, are obtained, then the point on the target object corresponding to the first camera pixel P(l1, l9) is illuminated twice: when projection image 1 is projected, it is illuminated by the light (directly or reflected) at x1, and when projection image 2 is projected, it is illuminated by the light (directly or reflected) at x2.

[0073] The first camera pixel corresponding to only one of the light rays corresponding to the first coordinate is directly illuminated, while the light rays corresponding to the other first coordinates are reflected. For example, the point on the target object corresponding to the first camera pixel P(l1, l9) is directly illuminated by the light ray at x1 when projecting image 1, and is illuminated by the light ray at x2 after reflection when projecting image 2.

[0074] An object of the present disclosure is to determine, among a plurality of second camera pixels, a second camera pixel that corresponds to the same point on the target object as a first camera pixel.

[0075] Reference Figure 1 Point Q1 of the target object is located at a first camera pixel corresponding to the first camera 12. The first camera pixel is illuminated twice, namely, by the single-reflection light d1 of the light a1 and the multiple-reflection light c1 of the light a2. The first first coordinate corresponding to the light a1 and the second first coordinate corresponding to the light a2 can be determined. A corresponding second camera pixel can be determined based on the first first coordinate, and a corresponding second camera pixel can be determined based on the second first coordinate. Multiple second camera pixels can be determined for each first camera pixel.

[0076] S503 : Determine, from a plurality of second camera pixels of a second camera, a first number of second camera pixels corresponding to a first number of first projection pixels according to a correspondence between the plurality of frames of second images and the plurality of frames of projection images.

[0077] In the embodiment of the present disclosure, the first phase corresponding to the first coordinate can be determined according to the following expression (1):

[0078]

[0079] In the above expression (1), px represents a coordinate (such as the first coordinate), Φ(px) represents the phase corresponding to the coordinate px (such as the first phase corresponding to the first coordinate), and f is the spatial frequency of the sinusoidal fringes used when calibrating the imaging system, that is, the number of cycles of the projected image, as shown in Figure 3Where f=6, W represents the number of optomechanical pixels included in each frame of the projected image, such as Figure 3 In the example, W is 30.

[0080] In the embodiment of the present disclosure, the mapping function of the imaging system calibration is rx=f(lx, ly, lΦ q ), where f() represents the mapping function, lx represents the horizontal coordinate of the first camera pixel, ly represents the vertical coordinate of the first camera pixel, and lΦ q represents the first phase corresponding to the first coordinate q. rx represents the abscissa of the second camera pixel corresponding to the first coordinate q, wherein, since epipolar correction has been performed in the pre-calibration process, the ordinate of the second camera pixel ry=ly, then for the first number (Q) of first phases (lΦ1, lΦ2, ..., lΦ Q ), correspondingly, multiple second coordinates can be obtained and expressed as P(rx, ry).

[0081] S504 : Decode the second camera pixels according to the correspondence between the multiple frames of second images and the multiple frames of projection images to obtain multiple second coordinates of each second camera pixel in the first number of second camera pixels under the projection light machine.

[0082] In the embodiment of the present disclosure, the method of determining the plurality of second coordinates may refer to S202 and will not be described in detail here.

[0083] Wherein, referring to the above, for the first camera pixel P(lx, ly), multiple second camera pixels are determined to be the second camera pixel P2(1), the second camera pixel P2(2), ..., the second camera pixel P2(Q), where Q is 3, then the corresponding multiple second coordinates can be determined for the second camera pixel P2(1), the corresponding multiple second coordinates can be determined for the second camera pixel P2(2), and the corresponding multiple second coordinates can be determined for the second camera pixel P2(Q).

[0084] S504 : Determine, according to the plurality of first coordinates and the plurality of second coordinates, second camera pixels matching the first camera pixels from a first number of second camera pixels.

[0085] In some embodiments, determining, based on a plurality of first coordinates and a plurality of second coordinates, a second camera pixel that matches a first camera pixel from a first number of second camera pixels includes: for each first coordinate from the plurality of first coordinates, determining that the second camera pixel corresponding to the first coordinate is a third camera pixel; determining a first matching value between each second coordinate from the plurality of second coordinates corresponding to the third camera pixel and the first coordinate; determining, from the plurality of first matching values ​​corresponding to the third camera pixel, a minimum first matching value as a second matching value for the third camera pixel; and determining, from the plurality of second matching values ​​corresponding to the first number of second camera pixels, that the second camera pixel corresponding to the third matching value is the second camera pixel that matches the first camera pixel, the third matching value being the minimum second matching value from the plurality of second matching values.

[0086] For example, for a first camera pixel P(lx, ly), the corresponding multiple first projected pixels can be determined to be first projected pixel t1 (first coordinate J1), first projected pixel t2 (first coordinate J2), ..., first projected pixel tQ (first coordinate JQ). Further, the multiple second camera pixels can be determined to be second camera pixel P2(1), second camera pixel P2(2), ..., second camera pixel P2(Q).

[0087] Then, for the first coordinate tq, the corresponding second camera pixel P2(q) is the third camera pixel, where the second camera pixel P2(q) corresponds to multiple second coordinates, namely, second coordinate K1, second coordinate K2, ..., second coordinate KG, where G is an integer greater than 1. Then, a first matching value between the second coordinate Kg and the first coordinate tq is determined, where g ranges from 1 to G. Furthermore, G first matching values ​​can be determined, such as first matching value 1, first matching value 2, ..., first matching value G.

[0088] Furthermore, the smallest first matching value (e.g., first matching value 2) among the G first matching values ​​is determined as the second matching value. Then, a corresponding second matching value can be determined for each second camera pixel P2(q). For example, the second camera pixel P2(1), the second camera pixel P2(2), ..., and the second camera pixel P2(Q) correspond to the second matching value 1, the second matching value 2, ..., and the second matching value Q, respectively.

[0089] Then, the smallest second matching value (such as the second matching value 3) among the Q second matching values ​​is determined as the third matching value. The third matching value (the second matching value 3) corresponds to the second camera pixel P2(3), and it is determined that the second camera pixel P2(3) and the first camera pixel P(lx, ly) have a matching relationship, that is, the second camera pixel P2(3) and the first camera pixel P(lx, ly) capture the same point of the target object.

[0090] The above method can be used to determine the second camera pixel that matches each first camera pixel.

[0091] In some embodiments, determining a first matching value between each of a plurality of second coordinates corresponding to a pixel of a third camera and a first coordinate includes: determining a first phase corresponding to the first coordinate; determining a plurality of second coordinates corresponding to the pixel of the third camera and a plurality of second phases corresponding thereto, wherein the plurality of second coordinates correspond one-to-one with the plurality of second phases; and determining, for each of the plurality of second phases, a matching value between the second phase and the first phase as a first matching value.

[0092] In the embodiment of the present disclosure, the first phase corresponding to the first coordinate and the second phase corresponding to the second coordinate can be determined according to the above expression (1).

[0093] Correspondingly, based on the above expression (1), each first coordinate can obtain a first phase, and then the multiple first coordinates corresponding to the first camera pixel can obtain multiple first phases. Similarly, based on the above expression (1), each second coordinate can obtain a second phase, and then the multiple second coordinates corresponding to the third camera pixel can obtain multiple second phases. It can further be determined that each first phase corresponds to multiple second phases.

[0094] For example, for a first camera pixel, the corresponding lΦ1, lΦ2, ..., lΦ can be determined. Q , a total of Q first phases. For the first phase lΦ q , we can determine the corresponding rΦ1, rΦ2,…, rΦ G , a total of G second phases.

[0095] The multiple first phases corresponding to the first camera pixels reflect the timing information of the first camera pixels being illuminated under different projection images, and the multiple second phases corresponding to the third camera pixels reflect the timing information of the third camera pixels being illuminated under different projection images.

[0096] Furthermore, the matching value of the first phase and the second phase can be determined using the following expression (2):

[0097] C qg = (lΦ q – rΦ g ) 2 (2)

[0098] Among them, in expression (2), C qg Indicates the first phase lΦ q and the second phase rΦ g The matching value of C qg Represents the difference between the two phases, C qg The smaller the value, the higher the matching degree.

[0099] S505 : For each first camera pixel among the plurality of first camera pixels, determine a disparity between the first camera pixel and a second target camera pixel.

[0100] The second target camera pixel is a second camera pixel that has a matching relationship with the first camera pixel.

[0101] In some embodiments, if the horizontal coordinate of the first camera pixel is lx, the horizontal coordinate of the second target camera pixel to be matched is rx. _ b, then the parallax can be determined using the following expression (3):

[0102] d=lx-rx_b (3)

[0103] In Expression (3), d represents a disparity, which indicates the horizontal displacement of the same point of the target object in the first image and the second image, and is used to calculate the depth of the same point.

[0104] In some embodiments, determining the disparity between a first camera pixel and a second target camera pixel includes: determining a first camera coordinate of the first camera pixel under the first camera; determining a second camera coordinate of the second target camera pixel under the second camera; determining a second target matching value corresponding to an adjacent second camera pixel of the second target camera pixel; correcting the second camera coordinate based on a third matching value corresponding to the second target camera pixel and a third matching value corresponding to the adjacent second camera pixel to obtain a third camera coordinate; and determining the disparity based on the first camera coordinate and the third camera coordinate.

[0105] The first camera coordinates of the first camera pixel under the first camera are (lx, ly), and the second camera coordinates of the second target camera pixel under the second camera are (rx_b, ry_b). The adjacent second camera pixels of the second target camera pixel have the same ordinates as the second target camera pixel and are adjacent in the horizontal coordinate direction. For example, if the number of adjacent second camera pixels is 2, then the horizontal coordinate of one adjacent second camera pixel is rx_b-1, and the horizontal coordinate of the other adjacent second camera pixel is rx_b+1. The third matching value corresponding to the adjacent second camera pixel (rx_b-1) is expressed as C-, and the third matching value corresponding to the adjacent second camera pixel (rx_b+1) is expressed as C + The third matching value corresponding to the second target camera pixel (rx_b) is denoted as C0.

[0106] Furthermore, the third matching value (C-, C0 and C + ) to correct the second camera coordinates (rx_b) to obtain the third camera coordinates of the second target camera pixel.

[0107] In some embodiments, the number of adjacent second camera pixels is two, and the second camera coordinates are corrected based on the third matching values ​​corresponding to the second target camera pixels and the third matching values ​​corresponding to the adjacent second camera pixels to obtain the third camera coordinates, including: determining a first difference between a sum of the third matching values ​​corresponding to each adjacent second camera pixel and twice the third matching value corresponding to the second target camera pixel; determining a second difference between the third matching values ​​corresponding to each adjacent second camera pixel; determining a quotient of the second difference and twice the first difference; and determining the sum of the quotient and the second camera coordinates as the third camera coordinates.

[0108] Specifically, the third camera coordinates can be determined using the following expression (4):

[0109] rx_s = rx_b+ (C- - C + ) / [2 × (C- - 2C0 + C + )] (4)

[0110] Where rx_s represents the horizontal coordinate of the third camera coordinate.

[0111] The present disclosure may also adopt other methods to correct the second camera coordinate rx_b, which is not limited in the present disclosure.

[0112] Furthermore, the parallax can be determined according to the following expression (5):

[0113] d = lx - rx_s (5)

[0114] In expression (5), d represents disparity, that is, the difference between the horizontal coordinates of the pixels corresponding to the same point of the target object in the first image and the second image, and the unit can be pixel.

[0115] The present disclosure adopts the above-mentioned method to correct the coordinates of the second camera, thereby improving the matching accuracy of the first camera pixels and the second camera pixels, and further improving the accuracy of disparity calculation.

[0116] S506 , performing three-dimensional reconstruction of the target object according to the parallax.

[0117] In the disclosure, the depth value of a point on the target object is determined according to the parallax, and specifically the following expression (6) is used to determine the depth value:

[0118]

[0119] In expression (6), Z represents the depth value, that is, the actual distance between the point on the target object and the first camera, F refers to the focal length of the first camera, expressed in mm / pixel, and B is the baseline length of the first camera and the second camera, that is, the distance between the optical center of the first camera and the optical center of the second camera, and the unit can be mm.

[0120] In summary, the present disclosure uses multiple frames of projected images to determine, for each first camera pixel in the first camera, multiple second camera pixels that the first camera pixel may match in the second camera. The matching value is then used to determine one of the second camera pixels that actually matches, thereby achieving more accurate three-dimensional reconstruction of the target object in scenes with multiple reflections.

[0121] Reference Figure 7 , is a structural block diagram of a 3D reconstruction device 70 provided by the present disclosure, and the 3D reconstruction device 70 specifically includes:

[0122] An acquisition module 71 is configured to acquire a plurality of first image frames and a plurality of second image frames, wherein the plurality of first image frames are acquired by a first camera capturing a target object when the projection machine projects the plurality of projection images onto the target object, and the plurality of second image frames are acquired by a second camera capturing the target object when the projection machine projects the plurality of projection images onto the target object, wherein each projection image frame in the plurality of projection image frames includes a plurality of projection pixels;

[0123] a first determining module 72 configured to determine, for each first camera pixel among a plurality of first camera pixels of the first camera, a first number of first projection pixels corresponding to the first camera pixel from a plurality of projection pixels included in the plurality of projection images based on the plurality of first images;

[0124] a second determining module 73 for determining, based on a correspondence between the plurality of frames of second images and the plurality of frames of projection images, a first number of second camera pixels corresponding to a first number of first projection pixels from a plurality of second camera pixels of the second camera, the first number of first projection pixels corresponding one-to-one to the first number of second camera pixels;

[0125] a third determining module 74 for determining, from the first number of second camera pixels, a second camera pixel that matches the first camera pixel;

[0126] The reconstruction module 75 is configured to perform three-dimensional reconstruction of the target object based on the matching relationship between the plurality of first camera pixels and the plurality of second camera pixels.

[0127] In an optional embodiment, the multiple frames of first images correspond to the multiple frames of projection images in a one-to-one manner, and determining, based on the multiple frames of first images, a first number of first projection pixels corresponding to the first camera pixels from among the multiple projection pixels included in the multiple frames of projection images, includes:

[0128] Decoding the first camera pixels according to the correspondence between the multiple frames of first images and the multiple frames of projection images to obtain multiple first coordinates corresponding to the first camera pixels under the projection light machine;

[0129] Each of the plurality of first coordinates is a coordinate of a corresponding first projection pixel under the projection light engine.

[0130] In an optional embodiment, the multiple frames of second images correspond to the multiple frames of projection images in a one-to-one manner, and the third determining module 74 is configured to decode the second camera pixels according to the correspondence between the multiple frames of second images and the multiple frames of projection images to obtain multiple second coordinates of each second camera pixel in the first number of second camera pixels under the projection light machine;

[0131] Second camera pixels matching the first camera pixels are determined among a first number of second camera pixels based on the plurality of first coordinates and the plurality of second coordinates.

[0132] In an optional embodiment, when determining, based on the plurality of first coordinates and the plurality of second coordinates, the second camera pixel matching the first camera pixel from the first number of second camera pixels, the third determining module 74 is specifically configured to:

[0133] For each first coordinate among the plurality of first coordinates, determining a second camera pixel corresponding to the first coordinate as a third camera pixel;

[0134] determining a first matching value between each second coordinate and the first coordinate in a plurality of second coordinates corresponding to a pixel of the third camera;

[0135] Determine, among the plurality of first matching values ​​corresponding to the third camera pixel, a minimum first matching value as the second matching value of the third camera pixel;

[0136] Among the plurality of second matching values ​​corresponding to the first number of second camera pixels, the second camera pixel corresponding to the third matching value is determined to be the second camera pixel matching the first camera pixel, and the third matching value is the smallest second matching value among the plurality of second matching values.

[0137] In an optional embodiment, when determining the first matching value between each second coordinate in the plurality of second coordinates corresponding to the third camera pixel and the first coordinate, the third determining module 74 is specifically configured to:

[0138] determining a first phase corresponding to the first coordinate;

[0139] Determine a plurality of second coordinates and a plurality of second phases corresponding to pixels of the third camera, wherein the plurality of second coordinates correspond to the plurality of second phases in a one-to-one manner;

[0140] For each second phase of the plurality of second phases, a matching value between the second phase and the first phase is determined to be a first matching value.

[0141] In an optional embodiment, the reconstruction module 75 is specifically configured to

[0142] determining, for each first camera pixel of the plurality of first camera pixels, a disparity between the first camera pixel and a second target camera pixel, the second target camera pixel being a second camera pixel having a matching relationship with the first camera pixel;

[0143] Based on the disparity, the target object is reconstructed in three dimensions.

[0144] In an optional embodiment, the reconstruction module 75, when determining the disparity between the first camera pixel and the second target camera pixel, is specifically configured to:

[0145] determining first camera coordinates of the first camera pixel under the first camera;

[0146] Determine the second camera coordinates of the second target camera pixel under the second camera;

[0147] determining a second target matching value corresponding to a second camera pixel adjacent to the second target camera pixel;

[0148] Correcting the second camera coordinates according to the third matching value corresponding to the second target camera pixel and the third matching value corresponding to the adjacent second camera pixel to obtain the third camera coordinates;

[0149] A disparity is determined based on the first camera coordinates and the third camera coordinates.

[0150] In an optional embodiment, the number of adjacent second camera pixels is two, and the reconstruction module 75, when correcting the second camera coordinates based on the third matching value corresponding to the second target camera pixel and the third matching value corresponding to the adjacent second camera pixel to obtain the third camera coordinates, is specifically configured to:

[0151] Determine a first difference between a sum of the third matching values ​​corresponding to each adjacent second camera pixel and twice the third matching value corresponding to the second target camera pixel;

[0152] Determine a second difference of the third matching values ​​corresponding to adjacent second camera pixels;

[0153] determining a quotient of the second difference and twice the first difference;

[0154] The sum of the quotient and the second camera coordinates is determined to be the third camera coordinates.

[0155] In an optional embodiment, multiple frames of projection images are projected onto the target object according to a preset timing, and each frame of projection image in the multiple frames of projection images includes: multiple optomechanical pixels, and the multiple optomechanical pixels include multiple projection pixels and multiple non-projection pixels; wherein, the coordinates of the multiple projection pixels included in different frames of projection images in the multiple frames of projection images are different in the multiple optomechanical pixels, and the sum of the multiple projection pixels included in each frame of projection image in the multiple frames of projection images is multiple optomechanical pixels.

[0156] The three-dimensional reconstruction device provided by the present disclosure can implement the above-mentioned position determination method. Please refer to the above for details and will not be repeated here.

[0157] In addition, in some of the processes in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or in parallel. They are only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the "second", "first", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit the "second" and "first" to be different types.

[0158] Figure 8 This is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the electronic device 80 includes: a processor 81, and a memory 82 communicatively connected to the processor 81, and the memory 82 stores computer-executable instructions.

[0159] Among them, the processor executes the computer-executable instructions stored in the memory to implement the position determination method provided by any of the above method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0160] An embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement any of the above methods.

[0161] An embodiment of the present disclosure also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes any of the above methods.

[0162] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0163] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0165] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present disclosure. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0166] Those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example for illustration. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the above functions. The specific working process of the above system can refer to the corresponding process in the above method embodiment, and will not be repeated here.

[0167] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0168] It should be understood that the present disclosure is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire multiple frames of first images and multiple frames of second images, wherein the multiple frames of first images are acquired by a first camera capturing the target object when a projection machine projects multiple frames of projection images onto the target object, and the multiple frames of second images are acquired by a second camera capturing the target object when the projection machine projects the multiple frames of projection images onto the target object, wherein each frame of the multiple frames of projection images includes: a plurality of projection pixels; For each first camera pixel among a plurality of first camera pixels of the first camera, determining, based on the plurality of frames of first images, a first number of first projection pixels corresponding to the first camera pixel from a plurality of projection pixels included in the plurality of frames of projection images; Determining, from a plurality of second camera pixels of the second camera, a first number of second camera pixels corresponding to the first number of first projection pixels based on a correspondence between the plurality of frames of second images and the plurality of frames of projection images, where the first number of first projection pixels corresponds one-to-one to the first number of second camera pixels; determining, among the first number of second camera pixels, second camera pixels that match the first camera pixels; The target object is three-dimensionally reconstructed according to the matching relationship between the plurality of first camera pixels and the plurality of second camera pixels.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The multiple frames of first images correspond to the multiple frames of projection images in a one-to-one manner, and determining, based on the multiple frames of first images and from a plurality of projection pixels included in the multiple frames of projection images, a first number of first projection pixels corresponding to the first camera pixels comprises: Decoding the first camera pixels according to the correspondence between the multiple frames of first images and the multiple frames of projection images to obtain multiple first coordinates corresponding to the first camera pixels under the projection light machine; Each of the plurality of first coordinates is a coordinate of a corresponding first projection pixel under the projection light engine.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The multiple frames of second images correspond to the multiple frames of projection images in a one-to-one manner, and determining, from the first number of second camera pixels, second camera pixels that match the first camera pixels includes: Decoding the second camera pixels according to the correspondence between the multiple frames of second images and the multiple frames of projection images to obtain multiple second coordinates of each second camera pixel in the first number of second camera pixels under the projection light engine; According to the plurality of first coordinates and the plurality of second coordinates, second camera pixels matching the first camera pixels are determined among the first number of second camera pixels.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The determining, according to the plurality of first coordinates and the plurality of second coordinates, second camera pixels matching the first camera pixels from the first number of second camera pixels comprises: For each first coordinate among the plurality of first coordinates, determining the second camera pixel corresponding to the first coordinate as a third camera pixel; determining a first matching value between each second coordinate in a plurality of second coordinates corresponding to the third camera pixel and the first coordinate; Determine, among the plurality of first matching values ​​corresponding to the third camera pixel, a minimum first matching value as the second matching value of the third camera pixel; Among the plurality of second matching values ​​corresponding to the first number of second camera pixels, the second camera pixel corresponding to a third matching value is determined to be the second camera pixel matching the first camera pixel, and the third matching value is the smallest second matching value among the plurality of second matching values.

5. The three-dimensional reconstruction method according to claim 4, characterized in that: Determining a first matching value between each second coordinate in a plurality of second coordinates corresponding to the third camera pixel and the first coordinate includes: determining a first phase corresponding to the first coordinate; Determine a plurality of second coordinates and a plurality of second phases corresponding to the third camera pixel, wherein the plurality of second coordinates correspond to the plurality of second phases in a one-to-one manner; For each second phase of the plurality of second phases, a matching value between the second phase and the first phase is determined to be the first matching value.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The performing three-dimensional reconstruction of the target object according to the matching relationship between the plurality of first camera pixels and the plurality of second camera pixels includes: determining, for each first camera pixel of the plurality of first camera pixels, a disparity between the first camera pixel and a second target camera pixel, the second target camera pixel being a second camera pixel having a matching relationship with the first camera pixel; The target object is three-dimensionally reconstructed according to the parallax.

7. The three-dimensional reconstruction method according to claim 6, characterized in that: The determining the disparity between the first camera pixel and the second target camera pixel includes: determining first camera coordinates of the first camera pixel under the first camera; determining second camera coordinates of the second target camera pixel under the second camera; determining a second target matching value corresponding to a second camera pixel adjacent to the second target camera pixel; Correcting the second camera coordinates according to the third matching value corresponding to the second target camera pixel and the third matching value corresponding to the adjacent second camera pixel to obtain third camera coordinates; The disparity is determined according to the first camera coordinates and the third camera coordinates.

8. The three-dimensional reconstruction method according to claim 7, characterized in that: The number of the adjacent second camera pixels is two, and the correcting the second camera coordinates according to the third matching value corresponding to the second target camera pixel and the third matching value corresponding to the adjacent second camera pixel to obtain the third camera coordinates includes: Determine a first difference between a sum of the third matching values ​​corresponding to the adjacent second camera pixels and twice the third matching value corresponding to the second target camera pixel; Determine a second difference of the third matching values ​​corresponding to the adjacent second camera pixels; Determine a quotient of the second difference and twice the first difference; The sum of the quotient and the second camera coordinates is determined as third camera coordinates.

9. The three-dimensional reconstruction method according to any one of claims 1 to 8, characterized in that: The multiple frames of projection images are projected onto the target object according to a preset timing, and each frame of projection image in the multiple frames of projection images includes: multiple optomechanical pixels, and the multiple optomechanical pixels include the multiple projection pixels and multiple non-projection pixels; wherein, the coordinates of the multiple projection pixels included in different frames of projection images in the multiple frames of projection images are different in the multiple optomechanical pixels, and the sum of the multiple projection pixels included in each frame of projection image in the multiple frames of projection images is the multiple optomechanical pixels.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the three-dimensional reconstruction method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the three-dimensional reconstruction method according to any one of claims 1 to 9 when executed by a processor.

Images