Modeling apparatus and method

Abstract

A modeling device and method. The device acquires a first image and a second image of an object under test taken from a viewing angle. The device extracts an outline of the object under test based on the first image. The device extracts an appearance of the object under test in the second image based on the outline. The device calculates a size of the object under test based on a plurality of reference points in the images. The device generates a three-dimensional model of the object under test based on the outline, the appearance, and the size of the object under test.

Description

Technical Field

[0001] This disclosure relates to a modeling apparatus and method, and more particularly to a modeling apparatus and method based on multiple images. Background Technology

[0002] With the increasing demand for smart manufacturing and smart factories, the need for digital twin systems has emerged, among which the virtualization of 3D models of the real environment is a necessary technology for building such systems. However, in the practice of factory manufacturing, due to the confidentiality of the processed products and the difficulty in obtaining them, it is difficult to directly obtain the model files of the processed products.

[0003] Therefore, providing a convenient and highly accurate modeling technology is a goal that the industry urgently needs to strive for. Summary of the Invention

[0004] To address the aforementioned problems, this disclosure proposes a modeling apparatus comprising a camera and a processor. The camera is used to capture a first image and a second image of a test object from a viewpoint. The processor is electrically connected to the camera and is used to perform the following operations: based on the first image, capturing a contour of the test object; based on the contour of the test object, capturing an appearance of the test object in the second image; calculating a dimension of the test object based on a plurality of reference points in the first image or the second image; and generating a three-dimensional model of the test object based on the contour, appearance, and dimension of the test object.

[0005] This disclosure also provides a modeling method applicable to an electronic device, the steps of which include: acquiring a first image and a second image of a test object captured from a viewpoint; capturing a contour of the test object based on the first image; capturing an appearance of the test object based on the contour in the second image; calculating a dimension of the test object based on a plurality of reference points in the first image or the second image; and generating a three-dimensional model of the test object based on the contour, the appearance, and the dimension of the test object.

[0006] It should be understood that the foregoing general description and the following specific description are merely exemplary and explanatory, and are intended to provide further explanation of the claimed disclosure. Attached Figure Description

[0007] To make the above and other objects, features, advantages and embodiments of this disclosure more apparent and understandable, the accompanying drawings are described below:

[0008] Figure 1 This is a schematic diagram of the modeling apparatus in the first embodiment of this disclosure;

[0009] Figure 2 This is a usage scenario diagram of the modeling device in some embodiments of this disclosure;

[0010] Figure 3 This is a schematic diagram of the backlight image in some embodiments of this disclosure;

[0011] Figure 4 This is a schematic diagram of the supplementary lighting image in some embodiments of this disclosure;

[0012] Figure 5 This is a schematic diagram of the adjusted backlight image in some embodiments of this disclosure;

[0013] Figure 6 This is a schematic diagram of the adjusted supplementary lighting image in some embodiments of this disclosure;

[0014] Figure 7 This is a schematic diagram illustrating the generation of a three-dimensional shape based on the dimensions of the object being measured in some embodiments of this disclosure;

[0015] Figure 8 This is a schematic diagram illustrating the generation of a three-dimensional model based on the appearance of the object under test in some embodiments of this disclosure; and

[0016] Figure 9 This is a flowchart of the modeling method in the second embodiment of this disclosure.

[0017] The annotations in the attached figures are explained as follows:

[0018] 1: Modeling device

[0019] 12: Processor

[0020] 14: Camera

[0021] 16: Backlight

[0022] 18: Positive light source

[0023] RP: Reference Point

[0024] OBJ1~OBJ4: Images of the object being measured

[0025] IMG1~IMG4: Images

[0026] MD1, MD2: 3D models

[0027] 200: Modeling Methods

[0028] S201~S205: Steps. Detailed Implementation

[0029] To make the description of this disclosure more detailed and complete, reference may be made to the accompanying drawings and the various embodiments described below, in which the same numbers represent the same or similar elements.

[0030] Please refer to Figure 1This is a schematic diagram of the modeling device 1 in the first embodiment of this disclosure. The modeling device 1 includes a processor 12 and a camera 14, wherein the processor 12 is electrically connected to the camera 14. The modeling device 1 is used to build a three-dimensional model of the object under test based on the image.

[0031] In some embodiments, processor 12 may include a central processing unit (CPU), a graphics processing unit (GPU), a multiprocessor, a distributed processing system, an application-specific integrated circuit (ASIC), and / or a suitable computing unit.

[0032] Camera 14 is used to capture an image of the object being measured. In some embodiments, camera 14 may be a camera for capturing planar images and / or a depth camera for capturing depth images.

[0033] In some embodiments, firstly, the modeling device 1 captures two images of the object to be measured from the same viewpoint and extracts the outline of the object from the first image. Secondly, based on the outline of the object, the modeling device 1 extracts the appearance of the object from the second image. Next, the modeling device 1 calculates the actual size of the object based on multiple reference points in the first or second image. Finally, the modeling device 1 can generate a three-dimensional model of the object based on its outline, appearance, and actual size.

[0034] Specifically, camera 14 captures a first image and a second image of an object under test from a viewpoint, wherein one of the first image and the second image contains a plurality of reference points, and processor 12 performs the following operations: based on the first image, captures a contour of the object under test; based on the contour of the object under test, captures an appearance of the object under test in the second image; calculates a size of the object under test based on the plurality of reference points in the first image or the second image; and generates a three-dimensional model of the object under test based on the contour, the appearance and the size of the object under test.

[0035] In some embodiments, in order to more easily capture the outline of the object under test, when capturing the first image, the modeling device 1 provides backlighting behind the object under test based on the viewpoint of the camera 14, so that the outline of the object under test in the first image is clearer.

[0036] Specifically, the modeling device 1 further includes a backlight 16, which is electrically connected to the processor 12 and used to project a backlight onto the camera 14. A placement space is formed between the backlight 16 and the camera 14 to accommodate the object under test. Correspondingly, the operation of capturing the first image of the object under test further includes: the processor 12 turning on the backlight to project the backlight; and the camera 14 capturing the first image of the object under test while the backlight is projecting the backlight.

[0037] In some embodiments, since the shooting environment may not have sufficient light, in order to more easily capture the appearance of the object under test, when taking the second image, the modeling device 1 provides supplementary light in front of the object under test based on the perspective of the camera 14, so that the appearance of the object under test in the second image is clearer.

[0038] Specifically, the modeling device 1 further includes a positive light source 18, which is electrically connected to the processor and used to project a supplementary light onto the object under test. Correspondingly, the operation of capturing the second image of the object under test further includes: the processor 12 turning on the positive light source 18 to project the supplementary light; and the camera 14 capturing the second image of the object under test while the positive light source 18 is projecting the supplementary light.

[0039] In some embodiments, the backlight 16 and / or the front light source 18 are light-emitting diodes or other light sources. In this embodiment, the backlight 16 and the front light source 18 are white light sources to enable the camera 14 to capture clearer images.

[0040] In some embodiments, since the camera 14 captures multiple reference points RP when taking images, and based on the known spacing between the reference points RP, the modeling device 1 can calculate the ratio of image pixels to actual distances (i.e., pixel distance) using the number of pixels between the reference points RP in the image. Accordingly, the modeling device 1 can calculate the actual size of the object being measured in the image.

[0041] Specifically, the operation of the processor 12 to calculate the size of the object under test further includes: calculating a pixel distance based on an actual distance between multiple reference points and a number of pixels between multiple reference points in the first image or the second image; and calculating the size of the object under test based on the pixel distance.

[0042] For specific embodiments of the modeling device 1, please refer to Figure 2 As shown in the figure, the object to be measured can be placed on a platform above the backlight 16 of the modeling device 1 (i.e., the placement space), and the camera 14 can capture images of it in the direction of the backlight 16. In addition, the platform above the backlight 16 is provided with two reference points RP as references for calculating the size of the object to be measured.

[0043] Accordingly, when capturing the first image, the processor 12 controls the backlight 16 to turn on the backlight and controls the camera 14 to capture the first image. Please refer to further details. Figure 3 This is a schematic diagram of the backlit image IMG1 in some embodiments of this disclosure. The backlit image IMG1 includes two reference points RP and the object under test image OBJ1. By turning on the backlight, the modeling device 1 can capture a high-contrast backlit image IMG1, enabling the processor 12 to more accurately capture the outline of the object under test.

[0044] Furthermore, the modeling device 1 calculates the contour of the object under test based on the backlit image IMG1, which includes converting the backlit image IMG1 into a grayscale image, calculating the image edges of the grayscale pixels, and then generating the contour of the object under test based on the image edges.

[0045] Specifically, the operation of the processor 12 to capture a contour of the object under test further includes: calculating a plurality of grayscale values ​​of a plurality of pixels in the first image; calculating an edge in the first image corresponding to the object under test based on the plurality of grayscale values; and generating the contour of the object under test based on the edge.

[0046] In other embodiments, the modeling device 1 can capture the contour of the object to be measured in many different ways. For example, the operation of the modeling device 1 to capture the contour of the object to be measured from the backlit image IMG1 may also include at least one of the following operations: (1) obtaining the region of interest of the object to be measured. (1) interest; ROI), (2) use Gaussian filter to reduce noise in the backlight image IMG1, (3) calculate pixel grayscale value and extract the darkest connected block, (4) reduce the region of interest to extract features, (5) calculate the number of image channels (colors), (6) calculate the size of the object image, (7) generate grayscale image and filter using channel threshold, (8) merge three-channel images and calculate the difference between regions, (9) color the pixels in the region, (10) adjust the image size and perform image overlay, (11) calculate the superposition of multiple regions, (12) reduce the image domain and retain the object region, (13) extract color edges, (14) perform morphological image processing and close operation to smooth the contour, (15) find connected regions, (16) fill the holes in the region, (17) obtain the region center, (18) filter the region according to the features and obtain the contour of the object.

[0047] On the other hand, when capturing the second image, the processor 12 controls the positive light source 18 to turn on the fill light and controls the camera 14 to capture the second image. Please refer to further details. Figure 4This is a schematic diagram of the supplementary lighting image IMG2 in some embodiments of this disclosure. The supplementary lighting image IMG2 includes two reference points RP and the object under test image OBJ2. By turning on the supplementary lighting, the modeling device 1 can capture the shape of the object under test, enabling the processor 12 to capture a clearer appearance of the object under test.

[0048] In some embodiments, before capturing the supplementary lighting image IMG2, the modeling device 1 determines whether to turn on the supplementary lighting based on the shooting environment.

[0049] Specifically, the operation of capturing the second image of the object under test further includes: the processor 12 calculating an ambient brightness based on a third image captured by the camera 14; and in response to the ambient brightness being higher than a brightness threshold, the processor 12 turning off the positive light source.

[0050] For example, before capturing the second image, camera 14 first captures an image of the environment, and processor 12 calculates the ambient brightness. If the ambient brightness is sufficient, the positive light source 18 is turned off when capturing the second image.

[0051] After acquiring the backlit image IMG1 and the supplementary lighting image IMG2, the modeling device 1 can extract the appearance of the object under test from the supplementary lighting image IMG2 based on the outline of the object under test extracted from the backlit image IMG1. Since the backlit image IMG1 and the supplementary lighting image IMG2 are taken from the same perspective and the object under test is placed in the same position, the modeling device 1 can directly extract the appearance of the object under test from the supplementary lighting image IMG2 at the position where the outline was extracted.

[0052] Specifically, the operation of the processor 12 to capture the appearance of the object under test further includes: cropping the second image based on the outline of the object under test in the first image to capture the appearance of the object under test.

[0053] In some embodiments, to facilitate subsequent image processing, after extracting the outline of the object under test from the backlight image IMG1, the modeling device 1 further translates and rotates the backlight image IMG1 to center the object under test image OBJ1 in the image. Please refer to the adjusted backlight image IMG3. Figure 5 The image of the object under test, OBJ3, is positioned in the center of the backlit image, IMG3.

[0054] Correspondingly, modeling device 1 calculates the transformation matrix for translation and rotation operations on the backlight image IMG1, and adjusts the supplementary lighting image IMG2 based on the transformation matrix to perform the same translation and rotation operations. Please refer to the adjusted supplementary lighting image IMG4. Figure 6 Similar to the backlit image IMG3, the object image OBJ4 was positioned in the center of the supplementary lighting image IMG4.

[0055] In this way, based on the adjusted supplementary lighting image IMG4, the modeling device 1 is able to capture the appearance of the object after it has been aligned.

[0056] Specifically, the operation of the processor 12 to capture the appearance of the object under test further includes: calculating a transformation matrix based on the outline of the object under test in the first image; adjusting the second image based on the transformation matrix; and cropping the adjusted second image based on the outline of the object under test to capture the appearance of the object under test.

[0057] In other embodiments, the modeling device 1 can generate the adjusted backlight image IMG3 in many different ways. For example, the operation of the modeling device 1 to adjust the backlight image IMG1 to obtain the backlight image IMG3 may also include at least one of the following operations: (1) calculating the minimum bounding area of ​​the backlight image as a rotation basis, (2) obtaining the connected region of the bounding box, (3) calculating the elliptical axis within the region to obtain the long and short sides and direction, (4) obtaining the center of the region, (5) calculating the aspect ratio, (6) generating a circular region and comparing the image level, (7) establishing a unit homogeneous two-dimensional transformation matrix, (8) calculating the rotation parameters of the two-dimensional homogeneous transformation matrix, (9) calculating the translation parameters of the two-dimensional homogeneous transformation matrix, and (10) performing an affine transformation on the image to center the center of the region of the object to be measured.

[0058] Next, the modeling device 1 calculates the height of the stereo model based on the depth information obtained by the camera 14.

[0059] Specifically, camera 14 is a depth camera, and the operation of processor 12 in generating the stereo model of the object under test further includes: generating depth information corresponding to the object under test based on the first image or the second image; and generating the stereo model of the object under test based on the depth information, the outline, the appearance, and the size of the object under test.

[0060] Please refer to Figure 7 The modeling device 1 combines the previously obtained outline and depth information of the object to be measured to generate a stereo model MD1, wherein the stereo model MD1 has the horizontal shape of the outline of the object to be measured and the height of the depth information.

[0061] Specifically, the operation of the processor 12 in generating the stereo model of the object under test further includes: generating a bottom surface of the stereo model based on the contour of the object under test; generating a thickness of the stereo model based on the depth information of the object under test; and constructing the stereo model based on the bottom surface and the thickness of the stereo model.

[0062] It should be noted that the depth information can be obtained from the first or second image captured by the depth camera 14, or it can be obtained from other images. Furthermore, in embodiments where the camera 14 is a planar camera, the depth information can be obtained from another depth camera, or calculated from images captured by multiple cameras with different viewpoints.

[0063] Finally, please refer to Figure 8 After establishing the stereo model MD1, the modeling device 1 further combines the stereo model MD1 with the previously obtained appearance of the object to be measured to generate a stereo model MD2, wherein the stereo model MD2 has the appearance of the surface of the object to be measured.

[0064] In summary, the modeling apparatus 1 proposed in this disclosure can extract the outline and appearance of the object from multiple images of the object being measured, and establish a three-dimensional model of the object accordingly. Furthermore, based on images taken under different light sources, the modeling apparatus 1 can more accurately extract the outline and clearer appearance of the object. Therefore, the modeling apparatus 1 provides a low-cost, high-efficiency, and high-precision modeling technique.

[0065] Please refer to Figure 9 This is a flowchart of the modeling method 200 in the second embodiment of this disclosure. The modeling method 200 includes steps S201 to S205. The modeling method 200 is used to establish a three-dimensional model of the object under test based on an image. The modeling method 200 can be executed by an electronic device (e.g., the modeling device 1 in the first embodiment).

[0066] First, in step S201, the electronic device acquires a first image and a second image of the object under test captured from a viewing angle, wherein one of the first image and the second image contains a plurality of reference points.

[0067] Next, in step S202, the electronic device captures a contour of the object under test based on the first image.

[0068] Next, in step S203, the electronic device captures an appearance of the object under test in the second image based on the contour.

[0069] Next, in step S204, the electronic device calculates a dimension of the object being measured based on multiple reference points in the first image or the second image.

[0070] Finally, in step S205, the electronic device generates a three-dimensional model of the object under test based on its outline, appearance, and size.

[0071] In some embodiments, step S201 further includes the electronic device turning on a backlight to project a backlight toward the viewing angle; and the electronic device capturing the first image of the object under test when the backlight is projected by the backlight.

[0072] In some embodiments, step S201 further includes the electronic device turning on a positive light source to project supplementary light toward the object under test; and the electronic device capturing the second image of the object under test when the positive light source projects the supplementary light.

[0073] In some embodiments, step S201 further includes the electronic device calculating an ambient brightness based on a third image; and in response to the ambient brightness being higher than a brightness threshold, the electronic device turning off the positive light source.

[0074] In some embodiments, step S202 further includes the electronic device calculating a plurality of grayscale values ​​of a plurality of pixels in the first image; the electronic device calculating an edge in the first image corresponding to the object to be measured based on the plurality of grayscale values; and the electronic device generating the contour of the object to be measured based on the edge.

[0075] In some embodiments, step S203 further includes the electronic device cropping the second image based on the outline of the object under test in the first image to capture the appearance of the object under test.

[0076] In some embodiments, step S203 further includes the electronic device calculating a transformation matrix based on the outline of the object under test in the first image; the electronic device adjusting the second image based on the transformation matrix; and the electronic device cropping the adjusted second image based on the outline of the object under test to capture the appearance of the object under test.

[0077] In some embodiments, step S204 further includes the electronic device calculating a pixel distance based on an actual distance between multiple reference points and a number of pixels between multiple reference points in the first image or the second image; and the electronic device calculating the size of the object under test based on the pixel distance.

[0078] In some embodiments, step S205 further includes the electronic device generating depth information corresponding to the object under test based on the first image or the second image; and the electronic device generating the three-dimensional model of the object under test based on the depth information, the outline, the appearance and the size of the object under test.

[0079] In some embodiments, step S205 further includes the electronic device generating a bottom surface of the three-dimensional model based on the contour of the object being measured; the electronic device generating a thickness of the three-dimensional model based on the depth information of the object being measured; and the electronic device constructing the three-dimensional model based on the bottom surface and the thickness of the three-dimensional model.

[0080] In summary, the modeling method 200 proposed in this disclosure can extract the contour and appearance of the object from multiple images of the object, and establish a three-dimensional model of the object accordingly. Furthermore, based on images taken under different light sources, the modeling method 200 can more accurately extract the contour and clearer appearance of the object. Therefore, the modeling method 200 provides a low-cost, high-efficiency, and high-accuracy modeling technique.

[0081] Although several embodiments have been described in detail above as examples, the modeling apparatus and method proposed in this disclosure can also be implemented in other systems, hardware, software, storage media, or combinations thereof. Therefore, the scope of protection of this disclosure should not be limited to the specific implementations described in the embodiments of this disclosure, but should be determined by the appended claims.

[0082] It will be apparent to those skilled in the art to which this disclosure pertains that various modifications and variations can be made to the structure of this disclosure without departing from its scope or spirit. In view of the foregoing, the scope of protection of this disclosure also covers modifications and variations made within the scope of the appended claims.

Claims

1. A modeling apparatus, comprising: A camera for capturing a first image and a second image of a measured object from a viewpoint, wherein one of the first image and the second image includes a plurality of reference points; and A processor, connected to the camera, is used to perform the following operations: Based on the first image, a contour of the object under test is extracted; Based on the outline of the object under test, an appearance of the object under test is captured in the second image; Calculate a dimension of the object under test based on a plurality of reference points in the first image or the second image; and Based on the outline, appearance, and size of the object under test, a three-dimensional model of the object under test is generated.

2. The modeling apparatus of claim 1, further comprising: A backlight is connected to the processor to project a backlight onto the camera, wherein a placement space is formed between the backlight and the camera to accommodate the object being measured; The operation of capturing the first image of the object under test further includes: The processor activates the backlight to project the backlight; and When the backlight is projected by the backlight source, the camera captures the first image of the object being measured.

3. The modeling apparatus of claim 1, further comprising: A positive light source, connected to the processor, is used to project supplementary light onto the object being tested; The operation of capturing the second image of the object under test further includes: The processor activates the positive light source to project the supplementary light; and When the supplementary light is projected by the positive light source, the camera captures the second image of the object being measured.

4. The modeling apparatus of claim 3, wherein the operation of capturing the second image of the object under test further comprises: The processor calculates an ambient brightness based on a third image captured by the camera; and In response to the ambient brightness exceeding a brightness threshold, the processor turns off the positive light source.

5. The modeling apparatus of claim 1, wherein the operation of capturing the contour of the object under test further comprises: Calculate the plurality of grayscale values ​​of a plurality of pixels in the first image; Based on the multiple grayscale values, calculate an edge in the first image corresponding to the object being measured; and Based on this edge, the contour of the object being measured is generated.

6. The modeling apparatus of claim 1, wherein the operation of capturing the appearance of the object under test further comprises: Based on the outline of the object under test in the first image, the second image is cropped to capture the appearance of the object under test.

7. The modeling apparatus of claim 1, wherein the operation of capturing the appearance of the object under test further comprises: Based on the contour of the object in the first image, a transformation matrix is ​​calculated; Based on this transformation matrix, adjust the second image; and Based on the outline of the object under test, the cropped and adjusted second image is used to capture the appearance of the object under test.

8. The modeling apparatus of claim 1, wherein the operation of calculating the dimension of the object under test further comprises: A pixel distance is calculated based on an actual distance between multiple reference points and the number of pixels between the multiple reference points in the first image or the second image; and The size of the object being measured is calculated based on the pixel distance.

9. The modeling apparatus of claim 1, wherein the camera is a depth camera, and the operation of generating the stereo model of the object under test further comprises: Based on the first image or the second image, depth information corresponding to the object being measured is generated; and Based on the depth information, contour, appearance, and size of the object under test, a three-dimensional model of the object under test is generated.

10. The modeling apparatus of claim 9, wherein the operation of generating the three-dimensional model of the object under test further comprises: Based on the contour of the object being measured, a bottom surface of the 3D model is generated; Based on the depth information of the object being measured, a thickness of the 3D model is generated; and The three-dimensional model is constructed based on its bottom surface and its thickness.

11. A modeling method applicable to an electronic device, comprising the steps of: A first image and a second image of a test object are acquired from a viewpoint, wherein one of the first image and the second image contains a plurality of reference points; Based on the first image, a contour of the object under test is extracted; Based on this contour, an appearance of the object under test is captured in the second image; Based on the multiple reference points in the first image or the second image, a dimension of the object under test is calculated; as well as Based on the outline, appearance, and size of the object under test, a three-dimensional model of the object under test is generated.

12. The modeling method of claim 11, wherein the step of capturing the first image of the object under test further comprises: Turn on a backlight to project a backlight toward that viewing angle; and When the backlight is projected by the backlight source, the first image of the object under test is captured.

13. The modeling method of claim 11, wherein the step of capturing the second image of the object under test further comprises: Turn on a positive light source to project a supplementary light onto the object being tested; and When the supplementary light is projected by the positive light source, the second image of the object under test is captured.

14. The modeling method of claim 13, wherein the step of capturing the second image of the object under test further comprises: Calculate an ambient brightness based on a third image; and The positive light source is turned off in response to the ambient brightness exceeding a brightness threshold.

15. The modeling method of claim 11, wherein the step of extracting the contour of the object under test further comprises: Calculate the plurality of grayscale values ​​of a plurality of pixels in the first image; Based on the multiple grayscale values, calculate an edge in the first image corresponding to the object being measured; and Based on this edge, the contour of the object being measured is generated.

16. The modeling method of claim 11, wherein the step of capturing the appearance of the object under test further comprises: Based on the outline of the object under test in the first image, the second image is cropped to capture the appearance of the object under test.

17. The modeling method of claim 11, wherein the step of capturing the appearance of the object under test further comprises: Based on the contour of the object in the first image, a transformation matrix is ​​calculated; Based on this transformation matrix, adjust the second image; and Based on the outline of the object under test, the cropped and adjusted second image is used to capture the appearance of the object under test.

18. The modeling method of claim 11, wherein the step of calculating the dimension of the object under test further comprises: A pixel distance is calculated based on an actual distance between multiple reference points and the number of pixels between the multiple reference points in the first image or the second image; and The size of the object being measured is calculated based on the pixel distance.

19. The modeling method of claim 11, wherein the step of generating the three-dimensional model of the object under test further comprises: Based on the first image or the second image, depth information corresponding to the object being measured is generated; and Based on the depth information, contour, appearance, and size of the object under test, a three-dimensional model of the object under test is generated.

20. The modeling method of claim 19, wherein the step of generating the three-dimensional model of the object under test further comprises: Based on the contour of the object being measured, a bottom surface of the 3D model is generated; Based on the depth information of the object being measured, a thickness of the 3D model is generated; and The three-dimensional model is constructed based on its bottom surface and its thickness.

Images