Three-dimensional reconstruction method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a three-dimensional reconstruction method and device, electronic equipment, a storage medium and a program product. The three-dimensional reconstruction method comprises the steps that shot images and point cloud images of a detection object in different view fields are acquired; based on the conversion relation, first position information included in the shot image and second position information included in the point cloud image are processed respectively, and multiple pieces of reconstruction position information are obtained, the conversion relation is determined according to a view field range of an image acquisition device used for shooting the image or distance information of multiple pieces of second position information in the point cloud image; and performing three-dimensional reconstruction on the detection object based on the plurality of pieces of reconstruction position information.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing, point cloud processing and industrial technology, and more specifically, to a three-dimensional reconstruction method, device, electronic device, storage medium and program product. Background Art

[0002] With the development of industrial technology, 3D model data is increasingly used in various industrial fields. 3D reconstruction refers to the process of restoring the geometric structure of a 3D scene or detected object from captured images, point cloud images or other observation data. The goal of 3D reconstruction is to create a 3D model, which can be a wireframe model, surface model or voxel model.

[0003] Therefore, how to achieve more effective three-dimensional reconstruction of the detection object to ensure the accuracy of the obtained three-dimensional model data is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present disclosure provides a 3D reconstruction method, device, electronic device, storage medium and program product.

[0005] According to one aspect of the present disclosure, a three-dimensional reconstruction method is provided, including: acquiring captured images and point cloud images of a detection object in multiple fields of view; based on a conversion relationship, processing at least one of the first position information included in the above-mentioned captured images and the second position information included in the above-mentioned point cloud images to obtain multiple reconstructed position information, the above-mentioned conversion relationship is determined according to the field of view range of the image acquisition device used for the above-mentioned captured images, or the distance information of each of the multiple pieces of the above-mentioned second position information in the above-mentioned point cloud images; and, based on the above-mentioned multiple pieces of reconstructed position information, performing three-dimensional reconstruction of the above-mentioned detection object.

[0006] According to another aspect of the present disclosure, a three-dimensional reconstruction device is provided, including: an acquisition module, used to acquire captured images and point cloud images of a detection object in multiple fields of view; a processing module, used to process at least one of the first position information included in the above-mentioned captured images and the second position information included in the above-mentioned point cloud images based on a conversion relationship to obtain multiple reconstructed position information, wherein the above-mentioned conversion relationship is determined according to the field of view range of the image acquisition device used for the above-mentioned captured images, or the distance information of each of the multiple pieces of the above-mentioned second position information in the above-mentioned point cloud images; and a three-dimensional reconstruction module, used to perform three-dimensional reconstruction of the above-mentioned detection object based on the above-mentioned multiple pieces of reconstructed position information.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described in the present disclosure.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the processor implements the method described in the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided. The computer program product includes computer executable instructions. When the computer executable instructions are executed, they are used to implement the method described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0011] Figure 1 The system architecture to which the three-dimensional reconstruction method can be applied according to an embodiment of the present disclosure is schematically shown;

[0012] Figure 2 A flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 3 A schematic diagram schematically shows an example of a process of processing the first position information included in the captured image based on the conversion relationship to obtain reconstructed position information according to an embodiment of the present disclosure;

[0014] Figure 4A to Figure 4C An exemplary schematic diagram schematically illustrates a first projection angle according to an embodiment of the present disclosure;

[0015] Figure 5 An example schematic diagram schematically shows a target distance information determination process according to an embodiment of the present disclosure;

[0016] Figure 6A~Figure 6B A schematic diagram schematically shows an example of a process of processing the second position information included in the point cloud image based on the conversion relationship to obtain reconstructed position information according to an embodiment of the present disclosure;

[0017] Fig. 7A An example schematic diagram of a data acquisition system according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 7BAn example schematic diagram of a process of performing three-dimensional reconstruction on a detection object based on multiple reconstruction position information according to an embodiment of the present disclosure is schematically shown;

[0019] Fig. 8A An example schematic diagram schematically shows a processed brightness image according to an embodiment of the present disclosure;

[0020] Figure 8B An example schematic diagram schematically shows a processed captured image according to an embodiment of the present disclosure;

[0021] Fig. 9 A block diagram schematically shows a three-dimensional reconstruction device according to an embodiment of the present disclosure; and

[0022] Fig.10 A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In one example, the three-dimensional reconstruction method may include at least one of the following: a three-dimensional reconstruction method based on visual geometry and a three-dimensional reconstruction method based on deep learning.

[0029] The three-dimensional reconstruction method based on visual geometry refers to estimating the camera pose of the collected data through multi-field-of-view images, and then extracting features from the images and performing comparison and stitching to achieve the conversion from two-dimensional images to three-dimensional models.

[0030] The 3D reconstruction method based on deep learning refers to the use of the learning and fitting capabilities of deep neural networks to achieve the conversion from two-dimensional images to three-dimensional models.

[0031] However, since the three-dimensional model data obtained based on the above method only includes three-dimensional position information but not texture information, that is, the accuracy of the three-dimensional model data is relatively limited, how to more effectively realize three-dimensional reconstruction of the detection object to ensure the accuracy of the obtained three-dimensional model data is an urgent problem to be solved.

[0032] To this end, the present disclosure provides a three-dimensional reconstruction method including: acquiring captured images and point cloud images of a detection object in multiple fields of view; based on a conversion relationship, processing at least one of the first position information included in the captured image and the second position information included in the point cloud image to obtain multiple reconstructed position information, wherein the conversion relationship is determined based on the field of view range of an image acquisition device used to capture the image, or the distance information of each of the multiple second position information in the point cloud image; and, based on the multiple reconstructed position information, performing three-dimensional reconstruction of the detection object.

[0033] Figure 1 The system architecture to which the 3D reconstruction method can be applied according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between different devices.

[0035] It should be noted that the 3D reconstruction method provided in the embodiment of the present disclosure can generally be executed by the server 105 . Accordingly, the 3D reconstruction device provided in the embodiment of the present disclosure can generally be set in the server 105 .

[0036] Alternatively, the 3D reconstruction method provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the 3D reconstruction apparatus provided in the embodiment of the present disclosure may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0039] The above describes the system architecture that can be used for the three-dimensional reconstruction method provided by the present disclosure. Figure 2 For example, the three-dimensional reconstruction process of the present disclosure is further described.

[0040] Figure 2 The flowchart of the three-dimensional reconstruction method according to the embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the 3D reconstruction method 200 includes operations S210 to S230.

[0042] In operation S210 , captured images and point cloud images of the detection object in multiple fields of view are acquired.

[0043] In operation S220, based on the conversion relationship, the first position information included in the captured image and the second position information included in the point cloud image are processed separately to obtain multiple reconstructed position information. The conversion relationship is determined according to the field of view range of the image acquisition device used to capture the image, or the distance information of each of the multiple second position information in the point cloud image.

[0044] In operation S230 , three-dimensional reconstruction is performed on the detection object based on the plurality of reconstruction position information.

[0045] The detection object may refer to a target or object that needs to be paid attention to during the three-dimensional reconstruction process. The method of acquiring captured images and point cloud images can be configured according to actual business needs and is not limited here. For example, the method of acquiring captured images may include at least one of the following: an acquisition method based on a digital camera, an acquisition method based on scanning, and an acquisition method based on video capture. Alternatively, the method of acquiring point cloud data may include at least one of the following: an acquisition method based on laser radar, an acquisition method based on structured light projection, and an acquisition method based on stereo matching of binocular or multi-cameras, etc.

[0046] The captured image may refer to a two-dimensional image or photo captured by a camera, and the captured image may include data consisting of a series of first position information on the surface of the target or object. The captured image includes multiple points. The first position information may represent the position of each point in the two-dimensional plane relative to a certain reference origin. For example, the first position information may include coordinate values ​​corresponding to the X coordinate and the Y coordinate.

[0047] The point cloud image may refer to data consisting of a series of second position information on the surface of the target or object. The point cloud image represents the external shape, structure and geometric information of the detected object. The point cloud image includes multiple points. The second position information may represent the position of each point relative to a reference origin in three-dimensional space. For example, the second position information may include coordinate values ​​corresponding to the X coordinate and the Y coordinate respectively. The distance information of the second position information may include a coordinate value corresponding to the Z coordinate.

[0048] After obtaining the captured image and the point cloud image, at least one of the first position information included in the captured image and the second position information included in the point cloud image can be processed based on the conversion relationship to obtain multiple reconstructed position information. The reconstructed position information can refer to the position in the three-dimensional space. The method for obtaining the reconstructed position information can be configured according to actual business needs and is not limited here.

[0049] For example, the method of processing the first position information to obtain the reconstructed position information may include at least one of the following: a determination method based on depth estimation, a determination method based on feature matching, and a determination method based on bundle adjustment. The determination method based on depth estimation refers to the use of monocular vision technology to estimate depth information from images captured under multiple fields of view. The determination method based on feature matching refers to identifying and matching feature points in images captured under different fields of view, and using triangulation to determine the position in three-dimensional space. The determination method based on bundle adjustment may refer to combining images captured under multiple fields of view and known camera parameters, and using bundle adjustment to obtain reconstructed position information.

[0050] Alternatively, the method of processing the second position information to obtain the reconstructed position information may include at least one of the following: a method based on point cloud registration, a method based on point cloud feature extraction, and a method based on point cloud surface reconstruction. The method based on point cloud registration refers to registering point cloud images under different fields of view and aligning them to the same coordinate system to obtain the reconstructed position information. The method based on point cloud feature extraction refers to extracting features from point cloud images and determining the reconstructed position information based on the features of point cloud images under different fields of view. The method based on point cloud surface reconstruction refers to fitting the second position information through spherical harmonics or moving least squares (MLS) to obtain the reconstructed position information.

[0051] In one example, the conversion relationship may be determined according to the field of view of an image acquisition device used to capture the image. The field of view may refer to the observation angle or field of view of the image acquisition device when acquiring the captured image. In this case, based on the field of view of the image acquisition device, an affine transformation, a perspective transformation, or other geometric transformation may be performed on the first position information to correct the first position information and obtain the reconstructed position information.

[0052] In another example, the conversion relationship may be determined based on the distance information of each of the plurality of second position information in the point cloud image. In this case, the second position information may be converted based on the distance information of each of the plurality of second position information to obtain the reconstructed position information.

[0053] After obtaining multiple reconstructed position information, the detection object can be reconstructed in three dimensions based on the multiple reconstructed position information, so as to combine the first position information and the second position information and use the information of multiple fields of view to reconstruct the detection object in three dimensions. Three-dimensional reconstruction refers to the process of restoring a three-dimensional spatial structure from a two-dimensional captured image or point cloud image.

[0054] The shape of the detection object can be configured according to actual business needs and is not limited here. For example, the shape of the detection object may include at least one of the following: a sphere, a cube, a cylinder, a cone, a polyhedron, and an irregular shape. In one example, the spherical detection object may include a workpiece with a spherical or spherical shape, which may be a complete sphere or a partial sphere, such as a spherical surface and a hemispherical surface.

[0055] According to the embodiments of the present disclosure, by acquiring captured images and point cloud images from multiple fields of view, more spatial information can be obtained, thereby improving the accuracy of subsequent three-dimensional reconstruction, reducing information loss due to field of view limitations, and enabling subsequent three-dimensional reconstruction results to be close to the real surface of the detection object in all directions. By flexibly establishing a conversion relationship by utilizing the field of view range of the image acquisition device and the distance information of each point cloud data, and processing the position information of each captured image and point cloud image based on the conversion relationship, it is possible to adapt to the needs of different scenes. On this basis, by performing three-dimensional reconstruction of the detection object based on multiple reconstructed position information, the three-dimensional reconstruction effect and the accuracy of the obtained three-dimensional model are improved.

[0056] In the embodiment of the present disclosure, the conversion relationship can be determined according to the field of view range of the image acquisition device used to capture the image. Figure 3 The process of processing the first position information included in the captured image to obtain the reconstructed position information is further described based on the conversion relationship determined based on the field of view range of the image acquisition device used to capture the image.

[0057] Figure 3 The diagram schematically shows an example of a process of processing the first position information included in a captured image based on a conversion relationship to obtain reconstructed position information according to an embodiment of the present disclosure.

[0058] In 300, taking the example that the conversion relationship is determined according to the field of view of the image acquisition device used to capture the image, the process of obtaining the reconstructed position information based on the first position information is explained. The reconstruction process refers to the process of converting the position coordinates in the two-dimensional plane into the position coordinates in the three-dimensional space.

[0059] like Figure 3 As shown, according to the relationship between longitude and latitude coordinates and Cartesian coordinates, the field of view of the image acquisition device in the width direction is, the field of view of the image acquisition device in the height direction is, the image resolution of the captured image 301 is, and the first position information of any point A0 in the captured image 301 is, then the value range of is, and the value range of is.

[0060] In one example, the first projection angle can be determined based on the first position information and the field of view of the image acquisition device. The first projection angle refers to the direction from the position of the image acquisition device to the point on the two-dimensional plane projected into the three-dimensional space. The first projection angle can describe the angle between the projection light and the target. The first projection angle can include an azimuth angle and a pitch angle. The pitch angle can refer to the angle between the positive direction of the Z axis to the position of the three-dimensional position information in the three-dimensional space and the XOY plane, usually in - / 2 to / 2, the pitch angle can be used to describe the height or depth of the three-dimensional position information relative to the XOY plane. The azimuth angle can refer to the angle between the positive direction of the Z axis and the projection of the three-dimensional position information on the XOY plane and the positive direction of the Z axis, usually between 0 and 2 The azimuth angle can be used to describe the position of three-dimensional position information on the horizontal plane.

[0061] In a specific example, according to the first position information and the field of view of the image acquisition device, the process of determining the first projection angle can be shown in the following formulas (1) and (2).

[0062]

[0063] in, Characterizing the height information in the image resolution of the captured image 301, represents the coordinate value corresponding to the y-axis in the first position information, Characterizes the pitch angle, Characterizing width information in the image resolution of the captured image 301, represents the coordinate value corresponding to the x-axis in the first position information, Characterize the azimuth.

[0064] After obtaining the first projection angle, the preset projection radius (Projection Radius) can be combined to determine the reconstructed position information according to the first projection angle. The preset projection radius refers to the distance from the center of the object to the spherical surface of the object. The preset projection radius can determine the scale of the projection in the three-dimensional space and the distance of the projection from the image acquisition device. The first projection angle (Projection Angle) and the preset projection radius are parameters used when reconstructing two-dimensional coordinates into three-dimensional coordinates, which define how to project points on a two-dimensional plane into three-dimensional space. For example, the first position information of any point A0 in the captured image 301 The reconstructed position information of point A1 in the corresponding three-dimensional space .

[0065] In a specific example, according to the first projection angle and the preset projection radius, the process of determining the reconstructed position information can be shown in the following formulas (3) to (5).

[0066]

[0067] in, Characterizes the preset projection radius, Represents the coordinate value of the reconstructed position information on the X-axis, Represents the coordinate value of the reconstructed position information in the height direction, Represents the coordinate value of the reconstructed position information on the Z axis.

[0068] After obtaining the reconstructed position information corresponding to each point A0 in the captured image 301, the detection object can be three-dimensionally reconstructed based on multiple reconstructed position information to obtain the reconstructed detection object 302. For example, in an application scenario where the detection object is a sphere, the captured image 301 may include a first position information set corresponding to the detection object obtained by using an image acquisition device, and correspondingly, the reconstructed detection object 302 may include a reconstructed position information set of the sphere corresponding to the detection object.

[0069] According to the embodiments of the present disclosure, by combining the first position information and the field of view of the image acquisition device, it is possible to better adapt to the visual characteristics of different scenes and ensure the accuracy of the first projection angle. On this basis, by flexibly determining the reconstructed position information according to the changed first projection angle and projection radius, the accuracy of the reconstructed position information is improved.

[0070] The above describes how to process the first position information included in the captured image based on the conversion relationship to obtain the reconstructed position information. Figure 4A to Figure 4C The method of obtaining the first projection angle is further described.

[0071] Figure 4A to Figure 4C An example schematic diagram of a first projection angle according to an embodiment of the present disclosure is schematically shown.

[0072] like Figure 4A As shown, in an application scenario where the detection object is a sphere under an ideal state, the angle between the points corresponding to two adjacent first position information is uniform and unchanged.

[0073] like Figure 4B As shown, in an application scenario where the detection object is a sphere in an actual situation, the angle between the points corresponding to two adjacent first position information increases as the absolute value of the coordinate value of the Y axis increases.

[0074] like Figure 4CAs shown, in the application scenario where the detection object is a sphere, the distance between two adjacent first position information can be determined according to the number of first position information and the field of view of the image acquisition device. Since the first position information is a point in a two-dimensional plane, the Euclidean distance formula can be used to calculate the distance between two adjacent first position information. On this basis, the first projection distance can be determined according to the distance between the two adjacent first position information. On this basis, the first projection angle is determined according to the first projection distance and the preset projection radius.

[0075] In a specific example, according to the first position information and the field of view of the image acquisition device, the process of determining the first projection angle can be shown in the following formula (6).

[0076]

[0077] in, Characterizes the distance between two adjacent first position information, Characterizing the number of first position information, Characterizes the projection distance.

[0078] According to the embodiments of the present disclosure, the first projection distance is determined based on the distance between two adjacent first position information. Since the distance between two adjacent first position information is determined based on the number of first position information and the field of view of the image acquisition device, the relative position of the detection object in the actual environment can be judged more accurately. On this basis, the first projection angle is determined by combining the first projection distance and the preset projection radius, and the first projection angle can be adapted to the change in the height direction by fitting the change law of the first projection angle, thereby improving the accuracy of the subsequent reconstruction of the position information.

[0079] In the embodiment of the present disclosure, the conversion relationship can also be determined according to the distance information of each of the plurality of second position information in the point cloud image. Figure 5 The process of determining the target distance information in this case is further described.

[0080] Figure 5 An example schematic diagram of a target distance information determination process according to an embodiment of the present disclosure is schematically shown.

[0081] In 500, the process of obtaining the reconstructed position information based on the second position information is described by taking the case where the detection object is a sphere and the conversion relationship is determined according to the respective distance information of a plurality of second position information in the point cloud image as an example.

[0082] When the three-dimensional reconstruction is performed based on the reconstructed position information obtained based on the first position information, the reconstructed detection object is a standard sphere, that is, the radius from the center of the sphere to each point on the sphere is the same. However, in actual applications, the detection object may not be a standard sphere, so there is a deviation between the reconstruction result of the three-dimensional reconstruction based on the first position information alone and the actual detection object. Therefore, the three-dimensional reconstruction based on the second position information can be introduced, and by calculating the real distance from the center of the sphere to each point on the sphere, the reconstruction result 504 of the three-dimensional reconstruction combined with the second position information is closer to the actual detection object.

[0083] like Figure 5 As shown, the point cloud image 501 includes M rows and N columns of second position information. M and N are both positive integers. The distance information corresponding to each point in the point cloud image 501 represents the real distance from the point cloud acquisition device of the point cloud image 501 to the surface of the detection object. Column 502 on the point cloud image 501 corresponds to a circle of data 503 on the detection object. For example, in an application scenario where the detection object is a sphere, the point cloud image 501 may include a set of second position information corresponding to the detection object obtained by using an image acquisition device, and correspondingly, a reconstruction result 504 of a three-dimensional reconstruction obtained based on the second position information may include a set of reconstructed position information of the sphere corresponding to the detection object.

[0084] In one example, the target distance information can be obtained by calibrating and calculating the characterization sphere. For example, the distance information with the largest value can be determined in the distance information of each of the N second position information in each row to obtain M candidate distance information to filter the noise of the distance information. The number of candidate distance information of each column can be determined, and the column whose number meets the predetermined condition is determined as the target column, which is the column with the largest value of distance information, so the target column can be considered as the column corresponding to the maximum value point.

[0085] On this basis, the target distance information can be determined according to the target column including M distance information. For example, the first mean of the M distance information can be determined, and the first mean is determined as the target distance information. Alternatively, the M distance information can be sorted according to the numerical relationship, and the distance information at the front predetermined position in the sorted M distance information is selected, and the second mean of the distance information at the front predetermined position is determined, and the second mean is determined as the target distance information.

[0086] According to the embodiments of the present disclosure, by respectively determining the distance information with the largest value in each row and performing column-by-column counting to screen and obtain the target column, it is possible to filter the noise in the point cloud image to accurately identify the target column, thereby improving the accuracy of the target column determination. On this basis, by determining the target distance information based on the distance information included in the target column, the accuracy of the target distance information is improved, which in turn helps to improve the accuracy of the subsequent reconstructed position information.

[0087] The above describes how to obtain target distance information. Figure 6A~Figure 6B The process of processing the second position information included in the captured image to obtain the reconstructed position information is further described based on the conversion relationship determined based on the distance information of each of the multiple second position information in the point cloud image.

[0088] Figure 6A~Figure 6B The diagram schematically shows an example of a process of processing the second position information included in a point cloud image based on a conversion relationship to obtain reconstructed position information according to an embodiment of the present disclosure.

[0089] Combination Fig. 6A and Figure 6B , taking the application scenario where the detection object is a sphere as an example, the process of obtaining the reconstructed position information is explained.

[0090] like Fig. 6A As shown, the second position information set corresponding to the detection object shown in the top view 601 of the point cloud image is a circle. For a standard sphere, the center of the sphere is The position is the rotation center of the rotating device where the detection object is placed, and the distance from the second position information corresponding to the target distance information to the sphere center can be used as the preset projection radius. According to the preset projection radius and the target distance information, the translation amount from the point cloud camera coordinates to the world coordinates can be determined, that is, the distance 602 from the reference distance of the point cloud camera to the sphere center.

[0091] like Figure 6B As shown, the second position information set corresponding to the detection object shown in the top view 603 of the point cloud image is a circle, and the second projection distance can be determined according to the distance 602 from the reference distance of the point cloud camera to the center of the sphere and the distance information of each second position information. According to the second position information, the second projection angle is determined. On this basis, each second position information can be transformed according to the second projection angle and the second projection distance to determine the actual coordinates corresponding to each point to obtain the reconstructed position information.

[0092] In a specific example, according to the respective distance information of the plurality of second position information in the point cloud image, the process of processing the second position information to obtain the reconstructed position information can be shown in the following formulas (7) to (12).

[0093]

[0094] in, Characterize the target distance information, Characterizes the projection distance, Characterize the second projection angle, and Characterizing the second position information, Represents the row value, Characterizes the column value, Represents the number of rows, Characterize the number of columns, Represents the actual distance between pixels on each line, The coordinate value representing the first direction, The coordinate value representing the second direction, The coordinate value representing the third direction.

[0095] According to the embodiments of the present disclosure, by determining the target distance information from multiple distance information, the morphological features of the detected object in space can be effectively captured. On this basis, by combining the second position information and the target distance information, the second projection angle and projection distance corresponding to the second position information can be accurately calculated, thereby improving the accuracy of the subsequent reconstruction of the position information.

[0096] It should be noted that the method for determining the conversion relationship in the embodiment of the present disclosure can be configured according to actual business needs and is not limited here.

[0097] For example, the conversion relationship can be determined based on the field of view of the image acquisition device used to capture the image. In this case, the conversion relationship can be determined as described above. Figure 3 As shown, the first position information included in the captured image can be processed based on the conversion relationship determined by the field of view range of the image acquisition device used to capture the image to obtain the reconstructed position information.

[0098] Alternatively, the conversion relationship may be determined based on the distance information of each of the plurality of second position information in the point cloud image. In this case, the conversion relationship may be determined as described above. Figure 6A~Figure 6B As shown, the second position information included in the captured image can be processed based on the conversion relationship determined by the distance information of each of the multiple second position information in the point cloud image to obtain the reconstructed position information.

[0099] Alternatively, the conversion relationship may be determined based on the field of view of an image acquisition device used to capture the image and respective distance information of a plurality of second position information in the point cloud image. In this case, the reconstructed position information may include third position information and fourth position information. The first position information included in the captured image may be processed based on the conversion relationship determined based on the field of view of the image acquisition device used to capture the image to obtain the third position information; the second position information included in the captured image may be processed based on the conversion relationship determined based on the respective distance information of a plurality of second position information in the point cloud image to obtain the fourth position information. The method for determining the third position information may refer to the method described above. Figure 3 The conversion relationship shown in FIG. 1 is determined based on the field of view range of the image acquisition device used to capture the image, and the first position information included in the captured image is processed. The fourth information can refer to the above Figure 6A~Figure 6B The process shown is to process the second position information included in the captured image based on the conversion relationship determined by the distance information of each of the multiple second position information in the point cloud image.

[0100] The above describes the method of processing the first position information based on the conversion relationship and the method of processing the second position information based on the conversion relationship. Fig. 7A The process of acquiring captured images and point cloud images under different fields of view is further explained.

[0101] Fig. 7A An example schematic diagram of a data acquisition system according to an embodiment of the present disclosure is schematically shown.

[0102] like Fig. 7A As shown, the data acquisition system may include an image acquisition device 701, a point cloud acquisition device 702 for point cloud images, a rotation device 703, and a clamping device 704. The image acquisition device 701 and the point cloud acquisition device 702 are spaced a predetermined distance apart. The field of view may be determined according to an angle of the image acquisition device 701 relative to the detection object 705 and an angle of the point cloud acquisition device 702 relative to the detection object 705.

[0103] The detection object 705 can be placed on the rotating device 703 via the clamping device 704. The rotating device 703 can be used to change the rotation direction of the detection object 705, and the clamping device 704 can be used to change the flipping direction of the detection object 705. Changing the rotation direction by the rotating device 703 and changing the flipping direction by the clamping device 704 is equivalent to changing the angle of the image acquisition device 701 and the point cloud acquisition device 702 relative to the detection object 705.

[0104] During the image acquisition process, the rotation device 703 and the clamping device 704 can be used to change the field of view, and the image acquisition device 701 and the point cloud acquisition device 702 can be used to acquire images of the detection object 705 under multiple fields of view to obtain captured images and point cloud images. For example, in an application scenario where the detection object is a sphere, the image acquisition device 701 can be used to acquire captured images of the detection object 705, and the point cloud acquisition device 702 can be used to acquire point cloud images of the detection object 705.

[0105] According to the embodiments of the present disclosure, based on a predetermined distance between an image acquisition device and a point cloud acquisition device, the acquisition of different fields of view is ensured by utilizing the angles of the image acquisition device and the point cloud acquisition device relative to the detection object. By acquiring captured images and point cloud images of different fields of view, a comprehensive analysis of the detection object can be performed, which helps to ensure the comprehensiveness and accuracy of subsequent three-dimensional reconstruction effects.

[0106] The above describes how to obtain the captured image and point cloud image. Figure 7B The process of performing three-dimensional reconstruction of the detected object based on multiple reconstruction position information is described.

[0107] Figure 7B An example schematic diagram of a process of performing three-dimensional reconstruction on a detection object based on multiple reconstruction position information according to an embodiment of the present disclosure is schematically shown.

[0108] like Figure 7B As shown in FIG. 1 , when the detection object is a sphere, in the process of stitching a complete sphere, since the minimum number of acquisitions required is related to the field of view range of the point cloud acquisition device 702, the field of view range of the point cloud acquisition device 702 is The 3D reconstruction process is explained by taking the minimum number of acquisitions as 3 as an example.

[0109] For example, the first circle data of the spherical detection object is collected by the point cloud acquisition device 702 to obtain a point cloud image 706 with a field of view of 0 degrees. After the detection object 705 is flipped 60 degrees by the clamping device 704, the second circle data of the spherical detection object is collected by the point cloud acquisition device 702 to obtain a point cloud image 707 with a field of view of 60 degrees. After the detection object 705 is flipped 60 degrees by the clamping device 704, the third circle data of the spherical detection object is collected by the point cloud acquisition device 702 to obtain a point cloud image 708 with a field of view of 120 degrees. After obtaining the point cloud image 706 with a field of view of 0 degrees, the point cloud image 707 with a field of view of 60 degrees, and the point cloud image 708 with a field of view of 120 degrees, the three point cloud images can be spliced ​​to obtain a complete sphere.

[0110] In one example, multiple reconstructed position information under the field of view can be processed based on the angle corresponding to the field of view to obtain annular point cloud data under the field of view. The annular point cloud data can refer to a set of point cloud data arranged in a ring in a plane or space. For example, the multiple reconstructed position information can be rotated according to the angle between the clamping device 704 and the point cloud acquisition device 702 to obtain the transformed initial annular point cloud data under the field of view. On this basis, the initial annular point cloud data can be flipped according to the flipping angle between the images to obtain the annular point cloud data under the field of view.

[0111] In another example, the annular point cloud data in each field of view may be spliced ​​based on the rotation matrix determined by the annular point cloud data in the field of view and the annular point cloud data in other fields of view to obtain the target point cloud data. The splicing process of the point cloud image may include a coarse matching process and a fine matching process. The coarse matching process may be used to obtain the initial pose, and the fine matching process may be used to improve the registration accuracy.

[0112] For example, the rough matching process may refer to directly splicing the annular point cloud data under multiple fields of view to obtain the initial point cloud data, wherein there are overlapping areas between the annular point cloud data under the multiple fields of view.

[0113] Alternatively, the fine matching process may refer to performing fine matching based on a preset algorithm according to the overlapping area features between the initial point cloud data obtained by coarse matching, and then optimizing the parameters of the preset algorithm for the spherical point cloud data after fine matching to achieve the best matching effect.

[0114] In one example, for the reconstructed position information obtained based on the first position information, a first preset method designed for surface color information can be used for fine matching. The stitching result obtained based on the first position information is a standard spherical point cloud with texture. In another example, for the reconstructed position information obtained based on the second position information, a second preset method designed for point cloud position information can be used for fine matching. The stitching result obtained based on the second position information is a spherical point cloud with both real spherical surface position information and texture information. The first preset algorithm and the second preset algorithm can be used to align multiple point cloud data sets into a common coordinate system to achieve three-dimensional reconstruction, scene modeling or other analysis tasks.

[0115] The first preset method and the second preset method can be configured according to actual business needs and are not limited here. For example, the first preset method and the second preset method can include at least one of the following: a method based on iterative closest point (ICP), a method based on point-to-plane, a method based on feature matching, and a method based on a machine learning model.

[0116] The iterative closest point based method refers to obtaining the reconstructed position information by iteratively finding the closest point pair and minimizing the distance between the point pairs. The point-to-plane based method refers to using the distance from the point to the target point cloud plane as an error metric to obtain the reconstructed position information. The feature matching based method refers to extracting features from point cloud images in different fields of view, and determining the correspondence between point clouds through feature matching to obtain the reconstructed position information. The machine learning model based method can refer to using a machine learning model to predict the transformation relationship between point clouds to obtain the reconstructed position information.

[0117] For point cloud image 706, point cloud image 707 and point cloud image 708, point cloud image 706, point cloud image 707 and point cloud image 708 can be downsampled respectively to reduce the data amount of each point cloud image to obtain intermediate point cloud image 1, intermediate point cloud image 2 and intermediate point cloud image 3.

[0118] In a specific example, the intermediate point cloud image 1 can be used as the original point cloud image, and the intermediate point cloud image 2 can be used as the target point cloud image. After processing based on the second preset algorithm, a first rotation matrix can be obtained, and the intermediate point cloud image 1 can be matrix transformed based on the first rotation matrix to obtain the matched point cloud image 1. The intermediate point cloud image 3 can be used as the original point cloud image, and the intermediate point cloud image 2 can be used as the target point cloud image. After processing based on the second preset algorithm, a second rotation matrix can be obtained, and the intermediate point cloud image 3 can be matrix transformed based on the second rotation matrix to obtain the matched point cloud image 3. On this basis, the intermediate point cloud image 2, the matched point cloud image 1 and the matched point cloud image 3 can be merged, and the third preset algorithm can be used to fuse the multi-field texture of the merged point cloud, so that the transition at the splicing is more natural and the splicing marks are eliminated.

[0119] In another specific example, after obtaining the matched point cloud image 1, the intermediate point cloud image 2 and the matched point cloud image 1 can be merged, and the intermediate point cloud image 3 is used as the original point cloud image, and the merged intermediate point cloud image 2 and the matched point cloud image 1 are used as the target point cloud image, and the third rotation matrix is ​​obtained after processing based on the second preset algorithm, and the intermediate point cloud image 3 is matrix transformed based on the third rotation matrix to obtain the matched point cloud image 3, so as to take into account the matching of the point cloud image 1 and the point cloud image 3. On this basis, the intermediate point cloud image 2, the matched point cloud image 1 and the matched point cloud image 3 can be merged, and the third preset algorithm can be used to fuse the multi-field texture of the merged point cloud, so that the transition at the splicing is more natural, and the splicing marks are eliminated, so as to realize the full-scale development of the sphere detection object, and the reconstructed three-dimensional model is closer to the real surface of the sphere detection object in the field of view effect.

[0120] After obtaining the target point cloud data, the target point cloud data can be texture mapped. Texture mapping refers to mapping a texture to the surface of a 3D model to increase the visual details and realism of the 3D model. For example, the texture includes color, pattern or material details. The texture mapping method can be configured according to actual business needs and is not limited here.

[0121] For example, since the point cloud acquisition device obtains the brightness image of the laser band while obtaining the point cloud image, the brightness image corresponds to the points in the point cloud image one by one. The brightness image includes at least one of the following: color information, brightness value, surface material, texture details, and ambient lighting. Based on the brightness image, the brightness information can be filled into the texture coordinate information of the target point cloud data.

[0122] Alternatively, the captured image and the brightness image of the same area can be aligned to obtain a processed captured image and a processed brightness image. Since the visible light texture information includes the color, pattern and detail features of the object surface within the visible light range, the visible light texture information in the processed captured image can be filled into the texture coordinate information of the target point cloud data.

[0123] According to the embodiments of the present disclosure, by processing the multiple reconstructed position information under each field of view separately, the annular point cloud data under different fields of view can be obtained, so that information with more angles and dimensions can be obtained, which is helpful to improve the accuracy and integrity of subsequent three-dimensional reconstruction. On this basis, by determining the rotation matrix based on different annular point cloud data and splicing the multiple annular point cloud data based on the rotation matrix, the fusion of the reconstructed position information of different fields of view can be achieved, the integrity and consistency of the point cloud data can be improved, and then more complete target point cloud data can be formed, and the accuracy of three-dimensional reconstruction can be improved.

[0124] The above describes how to perform 3D reconstruction of the detected object based on multiple reconstruction position information. Fig. 8A The process of obtaining the processed brightness image is further explained.

[0125] Fig. 8A An example schematic diagram of a processed brightness image according to an embodiment of the present disclosure is schematically shown.

[0126] like Fig. 8A As shown, when a point cloud image of a detection object in a certain field of view is obtained, a brightness image corresponding to the point cloud image is obtained at the same time. The brightness image can be cropped according to the predetermined position information to obtain a processed brightness image. The predetermined position information can be used to indicate the position of the processed brightness image in the brightness image. The processed brightness image can refer to a valid area in the brightness image, and the valid area refers to an area containing useful information for analyzing or enhancing the point cloud image.

[0127] In one example, the predetermined position information may include preset coordinates (x1, y1), preset width w, and preset length h. The preset coordinates (x1, y1) represent the coordinates of the upper left point of the effective area in the luminance image, the preset width w represents the width of the effective area based on the coordinates of the upper left point, and the preset length h represents the length of the effective area based on the upper left point.

[0128] According to the embodiments of the present disclosure, by combining the predetermined position information, the relative position of the processed brightness image in the entire brightness image can be accurately determined, thereby improving the accuracy of the processed brightness image and further improving the accuracy of subsequent three-dimensional reconstruction.

[0129] The above describes the process of obtaining the processed brightness image. Figure 8B The process of obtaining the processed captured image is further described.

[0130] Figure 8B An example schematic diagram of a processed captured image according to an embodiment of the present disclosure is schematically shown.

[0131] like Figure 8B As shown, after obtaining the captured image 802, the captured image 802 can be processed according to the processed brightness image 805 to obtain the initial transformed image 806. For example, the positional relationship between the captured image 802 and the processed brightness image 805 can be determined, and based on the positional relationship, the captured image 802 is transformed to obtain the initial transformed image 806. The imaging positional relationship refers to the relative position and direction of the captured image 802 and the processed brightness image 805 in space, and how they are related to each other. The transformation process may include at least one of a flip transformation, a scaling transformation, a cropping transformation, and a splicing transformation. The direction of the label 804 in the initial transformed image 806 is basically consistent with the direction of the label 803 in the processed brightness image 805; the size of the initial transformed image 806 and the processed brightness image 805 are consistent.

[0132] The initial transformed image 806 may have a label 804, and the processed brightness image 805 may have a label 803. The label may refer to a pixel label, which may be used to identify a group of pixels in an image that belong to the same region or object.

[0133] After obtaining the initial transformed image 806, the position of the label 803 in the processed brightness image 805 and the position of the label 804 in the initial transformed image 806 can be obtained. The method for obtaining the position of the label can be configured according to actual business needs and is not limited here. In one example, the processed brightness image 805 can be input into the target detection model to obtain the bounding box of the label 803 in the processed brightness image 805, and the initial transformed image 806 can be input into the target detection model to obtain the bounding box of the label 804 in the initial transformed image 806.

[0134] After obtaining the position of the tag 803 and the position of the tag 804, the initial transformed image 806 may be processed according to the position of the tag 803 and the position of the tag 804 to obtain the intermediate transformed image 807. For example, a scaling transformation parameter may be determined according to the position of the tag 803 and the position of the tag 804, and a scaling transformation may be performed on the initial transformed image 806 based on the scaling transformation parameter to obtain the intermediate transformed image 807.

[0135] In a specific example, the scaling transformation parameters may be determined as shown in the following formula (13).

[0136]

[0137] in, Characterize the scaling transformation parameters, The bounding box representing label 803, The bounding box representing label 804, Characterizes the width information of the initial transformed image 806 .

[0138] After the intermediate transformed image 807 is obtained, the position of the label 804 in the intermediate transformed image 807 may be obtained. In one example, the intermediate transformed image 807 may be input into the object detection model to obtain the bounding box of the label 804 in the intermediate transformed image 807.

[0139] According to the position of the label 803 and the position of the label 804 in the intermediate transformed image 807, the intermediate transformed image 807 is processed to obtain a processed captured image 808. For example, the position may include center point coordinates, and alignment parameters may be determined according to the center point coordinates of the label 803 and the center point coordinates of the label 804 in the intermediate transformed image 807, and the label 804 in the intermediate transformed image 807 is aligned with the label 803 in the processed brightness image 805 based on the alignment parameters to obtain a processed captured image 808.

[0140] In a specific example, the alignment parameters may be determined as shown in the following formulas (14) to (16).

[0141]

[0142] in, The center point coordinates of the characterization tag 803, The center point coordinates of the characterization tag 804, represents the first stitching region in the intermediate transformed image 807, represents the second stitching region in the intermediate transformed image 807, Characterize the alignment parameters, Characterizes the height information of the processed brightness image.

[0143] The first stitching area and the second stitching area can be vertically stitched. The effective area in the intermediate transformation image 807 is slightly larger than the effective area in the brightness image. After obtaining the first stitching area and the second stitching area, the geometric transformation between the two stitching areas can be determined, and the images of the two stitching areas are transformed based on the geometric transformation so that the first stitching area and the second stitching area are aligned and stitched in the vertical direction to obtain a processed captured image 808.

[0144] After obtaining the processed brightness image 805 and the processed captured image 808, since the two have been aligned, the mapping relationship between the second position information in the processed brightness image 805 and the first position information in the processed captured image 808 can be determined, and based on the mapping relationship, the texture information of each first position information in the processed captured image 808 is filled into the reconstructed position information corresponding to the second position information in the processed brightness image 805 to obtain the target point cloud data.

[0145] For example, in an application scenario where the detection object is a sphere, the detection object may be a workpiece with a spherical or spherical shape. The processed captured image 808 and the processed brightness image 805 may include a second position information set of the sphere or spherical approximation obtained by using a point cloud acquisition device, and the processed brightness image 805 may include a second position information set of the sphere or spherical approximation obtained by using a point cloud acquisition device. Correspondingly, the target point cloud data may include a reconstructed position information set corresponding to the detection object.

[0146] According to the embodiments of the present disclosure, by obtaining a processed captured image based on a preset processing method, a comprehensive and detailed optimization processing of the captured image is achieved, so that the processed captured image and the processed brightness image are aligned. On this basis, a mapping relationship is determined based on the position of the label in the processed captured image and the position of the corresponding label in the processed brightness image of the point cloud image, and the filling of texture information is achieved based on the mapping relationship, thereby improving the accuracy and quality of the target point cloud data.

[0147] Fig. 9 The block diagram of a 3D reconstruction apparatus according to an embodiment of the present disclosure is schematically shown.

[0148] like Fig. 9 As shown, the 3D reconstruction device 900 may include an acquisition module 910 , a processing module 920 and a 3D reconstruction module 930 .

[0149] The acquisition module 910 is used to acquire the captured images and point cloud images of the detection object in multiple fields of view.

[0150] The processing module 920 is used to process the first position information included in the captured image and the second position information included in the point cloud image respectively based on the conversion relationship to obtain multiple reconstructed position information. The conversion relationship is determined according to the field of view range of the image acquisition device used to capture the image, or the distance information of each of the multiple second position information in the point cloud image.

[0151] The three-dimensional reconstruction module 930 is used to perform three-dimensional reconstruction on the detection object based on multiple reconstruction position information.

[0152] According to the embodiments of the present invention, any one or more of the modules, or at least part of the functions of any one of them, can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0153] It should be noted that the 3D reconstruction device part in the embodiment of the present disclosure corresponds to the 3D reconstruction method part in the embodiment of the present disclosure. The description of the 3D reconstruction device part specifically refers to the 3D reconstruction method part, which will not be repeated here.

[0154] Fig.10 A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method according to an embodiment of the present disclosure is schematically shown. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0155] like Fig.10As shown, the computer electronic device 1000 according to the embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0156] In the RAM 1003, various programs and data necessary for the operation of the electronic device 1000 are stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004.

[0157] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0158] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the three-dimensional reconstruction method according to the embodiment of the present disclosure is implemented.

[0159] In the present disclosure, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0160] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the three-dimensional reconstruction method provided by the embodiment of the present disclosure.

[0161] When the computer program is executed by the processor 1001, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0162] According to the embodiments of the present disclosure, program codes for executing the computer programs provided by the embodiments of the present disclosure may be written in any combination of one or more programming languages.

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. It should also be noted that in some alternative implementations, the functions marked in the blocks may also occur in an order different from that marked in the accompanying drawings.

[0164] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A three-dimensional reconstruction method, comprising: Obtaining photographic images and point cloud images of the detection object in multiple fields of view; Based on a conversion relationship, first position information included in the captured image and second position information included in the point cloud image are processed respectively to obtain a plurality of reconstructed position information, wherein the conversion relationship is determined according to a field of view range of an image acquisition device used for the captured image or respective distance information of a plurality of second position information in the point cloud image; as well as Based on the multiple reconstruction position information, the detection object is three-dimensionally reconstructed.

2. The method according to claim 1, wherein: The reconstructed position information includes third position information corresponding to the first position information and fourth position information corresponding to the second position information; Based on the conversion relationship, the first position information included in the captured image and the second position information included in the point cloud image are processed respectively to obtain a plurality of reconstructed position information including: Processing the first position information according to the field of view of the image acquisition device to obtain the third position information; and The second position information is processed according to respective distance information of the plurality of second position information to obtain the fourth position information.

3. The method according to claim 2, wherein: The processing of the first position information according to the field of view of the image acquisition device to obtain the third position information includes: For each of the first position information, Determining a first projection angle according to the first position information and the field of view of the image acquisition device; and The third position information is determined according to the first projection angle.

4. The method according to claim 3, wherein: The first projection angle includes a pitch angle and an azimuth angle; The determining of the first projection angle according to the first position information and the field of view of the image acquisition device comprises: Determining the pitch angle according to the coordinate value of the first direction in the first position information and the field of view of the image acquisition device in the vertical direction; and The azimuth angle is determined according to the coordinate value of the second direction in the first position information and the field of view of the image acquisition device in the horizontal direction, wherein the first direction and the second direction are orthogonal.

5. The method according to claim 3, wherein: The determining, according to the first projection angle, the third position information comprises: The third position information is determined according to a preset projection radius and the first projection angle, wherein the preset projection radius represents a distance from a spherical center of the object to a spherical surface of the object.

6. The method according to claim 3, wherein: The determining of the first projection angle according to the first position information and the field of view of the image acquisition device comprises: Determining a first projection distance according to a distance between two adjacent first position information, wherein the distance between the two adjacent first position information is determined according to the number of the first position information and the field of view of the image acquisition device; and The first projection angle is determined according to the first projection distance.

7. The method according to claim 2, wherein: The processing of the second position information according to the respective distance information of the plurality of second position information to obtain the fourth position information comprises: For each of the second position information, determining a second projection angle and a second projection distance according to the second position information and target distance information determined based on the plurality of distance information; and The fourth position information is determined according to the second position information, the second projection angle, and the second projection distance.

8. The method according to claim 7, wherein: The point cloud image includes M rows and N columns of second position information, where M and N are both positive integers; The target distance information is obtained by: In each row, the distance information of the N second position information is determined to have the largest value, and M candidate distance information is obtained; Determine the number of candidate distance information for each column, and determine the column whose number meets a predetermined condition as the target column; as well as The target distance information is determined based on the M distance information of the target column.

9. The method according to claim 7, wherein: Determining the second projection angle and the second projection distance according to the second position information and the target distance information determined based on the plurality of distance information includes: determining the second projection angle according to the second position information; Determining intermediate distance information according to a preset projection radius and the target distance information, wherein the preset projection radius is used to represent the distance from the spherical center of the object to the spherical surface of the object; and The second projection distance is determined according to the target distance information and the intermediate distance information.

10. The method according to claim 7, wherein: The determining the fourth position information according to the second position information, the second projection angle, and the second projection distance includes: Determining a coordinate value in a first direction and a coordinate value in a third direction according to the second projection angle and the second projection distance; and A coordinate value of a second direction is determined according to the second position information, wherein the first direction, the second direction and the third direction are orthogonal to each other.

11. The method according to any one of claims 1 to 10, wherein The image acquisition device and the point cloud acquisition device for the point cloud image are separated by a predetermined distance, and the field of view is determined according to an angle of the image acquisition device relative to the detection object and an angle of the point cloud acquisition device relative to the detection object; The acquiring of captured images and point cloud images of the detection object in multiple fields of view comprises: The image acquisition device and the point cloud acquisition device are used to acquire the detection object placed on the rotating device with the multiple fields of view respectively, so as to obtain the captured image and the point cloud image.

12. The method according to claim 11, wherein: The three-dimensional reconstruction of the detection object based on the multiple reconstruction position information includes: Based on an angle corresponding to the field of view, processing a plurality of reconstructed position information under the field of view to obtain annular point cloud data under the field of view; and Based on the rotation matrix determined by the annular point cloud data in the field of view and the annular point cloud data in other fields of view, the annular point cloud data in each field of view is spliced ​​to obtain target point cloud data.

13. The method according to any one of claims 1 to 10, wherein The three-dimensional reconstruction of the detection object based on the multiple reconstruction position information includes: For each of the fields of view, Determining a mapping relationship based on a position of a tag in the processed captured image and a position of a corresponding tag in the processed brightness image of the point cloud image, wherein the mapping relationship represents a relationship between the first position information and the second position information; and Based on the mapping relationship, the texture information of each first position information in the processed captured image is filled into the reconstructed position information corresponding to the second position information in the processed brightness image to obtain the target point cloud data.

14. The method according to claim 13, wherein: The processed brightness image is determined according to predetermined position information, and the predetermined position information is used to indicate the position of the processed brightness image in the brightness image; The processed captured image is obtained by: Based on the positional relationship between the brightness image and the captured image, the captured image is transformed to obtain an initial transformed image, wherein the transformation process includes at least one of the following: flip transformation, crop transformation and splicing transformation; According to the position of the label in the initial transformed image and the position of the label in the processed brightness image, the initial transformed image is scaled to obtain an intermediate transformed image; and The position of the label in the intermediate transformed image is aligned with the position of the label in the processed brightness image to obtain the processed captured image.

15. The method according to claim 14, wherein: The scaling process is performed on the initial transformed image according to the position of the label in the initial transformed image and the position of the label in the processed brightness image to obtain the intermediate transformed image, comprising: determining scaling transformation parameters based on the position of the label in the initial transformed image and the position of the label in the processed brightness image; and The initial transformed image is scaled according to the scaling transformation parameters to obtain the intermediate transformed image.

16. The method according to claim 14, wherein: The step of aligning the position of the label in the intermediate transformed image with the position of the label in the processed brightness image to obtain the processed captured image comprises: determining alignment parameters based on the position of the label in the intermediate transformed image and the position of the label in the processed brightness image; and According to the alignment parameters, the position of the label in the intermediate transformed image is aligned with the position of the label in the processed brightness image to obtain the processed captured image.

17. The method according to claim 1, wherein: The detection object is a sphere.

18. A three-dimensional reconstruction device, comprising: An acquisition module is used to acquire photographic images and point cloud images of the detection object in multiple fields of view; a processing module, configured to process at least one of the first position information included in the captured image and the second position information included in the point cloud image based on a conversion relationship to obtain a plurality of reconstructed position information, wherein the conversion relationship is determined according to a field of view range of an image acquisition device used for capturing the image, or respective distance information of a plurality of pieces of the second position information in the point cloud image; as well as The three-dimensional reconstruction module is used to perform three-dimensional reconstruction on the detection object based on the multiple reconstruction position information.

19. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images