Apparatus and method for obtaining a registration error map representing the sharpness level of an image

Abstract

The present invention generally relates to an apparatus and method for obtaining a registration error map representing the sharpness level of an image. Many methods are known for determining the position of a camera relative to an object based on knowledge of a 3D model of the object and the intrinsic parameters of the camera. However, regardless of the vision servo technique used, there is no control in the image space, and the object may leave the camera's field of view during servoing. These methods use visual information of different attributes. A registration error map related to an image of an object of interest is proposed, which is generated by calculating the intersection of a refocusing plane obtained from a 3D model of the object of interest and a focal stack based on acquired four-dimensional light field data related to the object of interest.

Description

[0001] This application is a divisional application of the Chinese patent application "Device and Method for Obtaining a Registration Error Map Representing the Sharpness Level of an Image" (application number: 201680034570.X) with an application date of June 16, 2016. Technical Field

[0002] The present disclosure generally relates to devices and methods for obtaining a registration error map from four-dimensional light field data, which can be used in visual servo applications such as visual guidance or quality control of surfaces. Background Art

[0003] Visual servoing is a technique that uses visual feedback information to control the movement of an end user such as a robot (or in some cases, a human's movement). This visual feedback information is received from a visual sensor (such as a camera) connected to or embedded in the robot, or is displayed on the screen of a device belonging to the end user (such as a mobile phone or a tablet).

[0004] As disclosed in "Path Planning for 3D Visual Servoing: for a Wheeled mobile Robot", H. Mekki & M. Letaiel, IEEE 2013 International Conference on Individual and Collective Behaviours in Robotics, there are two methods in visual servoing. The first is image-based control or IBC, and the second is position-based control or PBC.

[0005] In IBC, the visual feedback is directly defined in the image. However, IBC has stability and convergence problems.

[0006] In PBC (also known as 3D visual servoing), the control error function is calculated in the Cartesian space, and image features are extracted from the image. An ideal model of the target is used to determine the position of the target relative to the camera frame. Many methods are known for determining the position of the camera relative to the target based on knowledge of the 3D or 2D model of the target and the intrinsic parameters of the camera. These methods use visual information with different attributes, such as points, lines, etc.

[0007] However, in either visual servoing technology, whether it is IBC or PBC, there is no control in the image space, and during servoing, the object may leave the camera's field of view, making it difficult to determine the relative pose of the camera and the camera. Summary of the Invention

[0008] According to a first aspect of the present invention, there is provided an apparatus for obtaining a map, referred to as a registration error map, which represents the sharpness levels of a plurality of pixels of an image. The apparatus includes: a processor configured to obtain a registration error map related to an image of an object of interest, the registration error map being generated by calculating an intersection of a refocusing plane obtained from a three-dimensional model of the object of interest and a focal stack based on acquired four-dimensional light field data related to the object of interest.

[0009] According to an embodiment of the present invention, the processor is configured to determine the refocusing plane by calculating a distance map of the three-dimensional model when the object of interest is set at a reference position.

[0010] According to an embodiment of the present invention, the processor is configured to calculate the intersection of the refocusing plane and the focal stack by determining, for each pixel of the image to be refocused, a refocusing distance from a predetermined refocusing plane corresponding to one of the images constituting the focal stack.

[0011] According to an embodiment of the present invention, the processor is configured to generate an appearance of the registration error map to be displayed based on information related to the sharpness levels of the pixels of the refocused image.

[0012] According to an embodiment of the present invention, the processor is configured to display the obtained registration error map on a display device of the apparatus.

[0013] Another aspect of the present invention relates to a method for obtaining a map, referred to as a registration error map, which represents the sharpness levels of a plurality of pixels of an image. The method includes: obtaining a registration error map related to an image of an object of interest, the registration error map being generated by calculating an intersection of a refocusing plane obtained from a three-dimensional model of the object of interest and a focal stack based on acquired four-dimensional light field data related to the object of interest.

[0014] According to an embodiment of the present invention, determining the refocusing plane includes calculating a distance map of the three-dimensional model when the object of interest is set at a reference position.

[0015] According to an embodiment of the present invention, calculating the intersection of the refocusing plane and the focal stack includes: determining, for each pixel of the image to be refocused, a refocusing distance from a predetermined refocusing plane corresponding to one of the images constituting the focal stack.

[0016] According to an embodiment of the present invention, the method includes: generating an appearance of the registration error map to be displayed based on information related to the sharpness levels of the pixels of the refocused image.

[0017] According to an embodiment of the present invention, the method includes: displaying the obtained registration error map on a display device.

[0018] Some of the processes implemented by the elements of the present disclosure may be computer-implemented. Accordingly, such elements may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a "circuit", "module", or "system". In addition, these elements may take the form of a computer program product embodied in any tangible expression medium, with computer-usable program code embodied in any tangible expression medium.

[0019] Since the elements of the present disclosure may be implemented in software, the present disclosure may be embodied as computer-readable code for providing to a programmable device on any suitable carrier medium. Tangible carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, tape devices, or solid-state storage devices. Transient carrier media may include signals such as electrical signals, electronic signals, optical signals, acoustic signals, magnetic signals, or electromagnetic signals, such as microwave or RF signals.

[0020] The objectives and advantages of the present disclosure will be realized and attained by the elements and combinations particularly pointed out in the claims.

[0021] It should be understood that the above general description and the following description are both exemplary and explanatory, and not restrictive of the claimed invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A Schematically represents a full optical camera,

[0023] Figure 1B Represents a multi-array camera,

[0024] Figure 2 Schematically shows a plan view of sensor regions arranged on a sensing surface of an image sensor of a light field camera,

[0025] Figure 3 Shows a schematic light field camera including an ideal perfect thin lens model,

[0026] Figure 4 Is a schematic block diagram showing an example of a device for obtaining a registration error map representing the level of blurriness of an image according to an embodiment of the present disclosure,

[0027] Figure 5 Is a flowchart for explaining a process for obtaining a registration error map representing the level of blurriness of an image according to an embodiment of the present disclosure,

[0028] Figure 6 Represents a distance map derived from a digital image,

[0029] Figure 7 represents a focal stack calculated from the acquired 4D light field data

[0030] Figure 8 shows a registration error map obtained by performing a process for obtaining a registration error map representing the blur level of an image according to an embodiment of the present disclosure

[0031] Figure 9 represents a registration error map according to an embodiment of the present disclosure Detailed Description

[0032] As will be understood by those skilled in the art, aspects of the principles of the present disclosure may be implemented as a system, method, or computer-readable medium. Accordingly, aspects of the principles of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects that is commonly referred to herein as a "circuit," "module," or "system." In addition, aspects of the principles of the present disclosure may take the form of a computer-readable storage medium. Any combination of one or more computer-readable storage media (a) may be used

[0033] By arranging a microlens array between the main lens and the sensor, the plenoptic camera can measure the amount of light propagating along each beam of light intersecting the sensor. The data acquired by such a camera is referred to as light field data. These light field data can be processed to reconstruct an image of the scene according to different viewpoints. The light field data can be used to generate a focal stack, which includes a set of images each having a different refocusing depth. As a result, the user can change the focus of the image. Compared with a traditional camera, the plenoptic camera can obtain additional light information components to achieve scene image reconstruction according to different viewpoints and refocusing depths through post-processing

[0034] Therefore, these specificities of the light field data can be used in the context of visual servo and visual guidance, where any additional information that helps to find the correct position of the object of interest is useful

[0035] Figure 1A is a diagram schematically showing a plenoptic camera 100. The light field camera is capable of recording four-dimensional (or 4D) light field data. The plenoptic camera 100 includes a main lens 101, a microlens array 102, and an image sensor 104

[0036] Figure 1B represents a multi-array camera 110. The multi-array camera 110 includes a lens array 112 and an image sensor 114

[0037] In as Figure 1AIn the example of the all-optical camera 100 shown, the main lens 101 receives light from an object (not shown in the figure) in the object field of the main lens 101 and causes the light to pass through the image field of the main lens 101. The microlens array 102 includes a plurality of microlenses 103 arranged in a two-dimensional array.

[0038] The data captured by the light field camera can be post-processed to reconstruct an image of the scene from different viewpoints. Since the light field camera can acquire a set of partial views of the same scene from slightly different viewpoints, an image with a customized focal plane can be created by combining these different partial views.

[0039] Figure 2 is a plan view schematically showing an example of a sensor region arranged on the sensing surface of the image sensor of the light field camera. As Figure 2 shown, the image sensor 200 includes a plurality of pixels 201 arranged in a two-dimensional array and receives light from an object through the microlens array 202. Each microlens 203 of the microlens array 202 has a lens property that guides light to a circular region 204 on the image sensor 200. The outer contour of the circular region 204 can represent the shape of the microlens image formed and acquired on the image sensor 200, which depends on the shape 203 of the microlens. Among all the pixels 201 on the image sensor 200, only the pixels 201 substantially located inside the circular region 204 contribute to imaging. In other words, the pixel region (or sensing region) of each pixel 201 that contributes to imaging is substantially located inside the circular region 204.

[0040] The image sensor 200 of the light field camera records an image including a set of 2D microlens images arranged within a two-dimensional (or 2D) image. Each microlens 203 of the microlens array 202 forms a microlens image represented by the circular region 204. The coordinates of the pixels 201 on the sensor 200 are indicated by an ordered pair (x, y) in the x-y coordinate system on the surface of the image sensor 200, as Figure 2 shown. The distance p is the distance between two consecutive microlens images. The microlens 203 is selected such that the distance p is greater than the size of the pixel 201. The distance w is the parallax distance between two consecutive microlens images. As Figure 2 shown, the microlens images are indicated by their respective coordinates (i, j) in the i-j coordinate system on the surface of the image sensor 200.

[0041] As described above, only the pixels 201 substantially located inside the circular region 204 receive light through the microlens 203. The inter-microlens space can be masked to prevent photons from penetrating outside the microlens 203. If the microlens 203 has a square shape and does not form an inter-microlens space, such masking is not required.

[0042] The center of the microlens image (i, j) is located at the coordinate (x i，j ,y i，j ) on the image sensor 200. θ represents the angle between the square array of pixels 201 and the square array of microlenses 204. The coordinates (x i，j ,y i，j ) can be obtained by considering (x 0，0 ,y 0，0 ) is derived from the following equation (1) for the pixel coordinates of the microlens image (0, 0):

[0043]

[0044] The distances p and w are given in pixels. They are converted into physical units of distance, such as meters, P and W, respectively, by multiplying by the pixel size δ (in meters): W = δw and P = δp. These distances depend on the characteristics of the light field camera.

[0045] Here, we will refer to Figure 3 To discuss exemplary light properties for a light field camera, Figure 3 A schematic light field camera including an ideal perfect thin lens model is shown.

[0046] Main lens 301 has a focal length F and an aperture Φ. Microlens array 302 includes microlenses 303 with a focal length f. The pitch of microlens array 302 is φ. Microlens array 302 is located at a distance D from main lens 301 and at a distance d from sensor 304. An object (not shown) is located at a distance z from main lens 301. The object is focused by main lens 301 at a distance z′ from main lens 301. Figure 3 The case of D>z′ is shown. In this case, the microlens image can be focused on the sensor 304 according to d and f.

[0047] The parallax W varies with the distance z between the object and the main lens 301. The relationship between W and z is established by relying on the thin lens equation (2) and Thales' law equation (3)

[0048]

[0049]

[0050] Then, by combining equations (2) and (3), the following equation (4) is derived:

[0051]

[0052] The relationship between W and z does not assume that the microlensed image is in focus. The microlensed image is strictly in focus according to the following thin lens equation:

[0053]

[0054] The main attribute of the light field camera is the ability to compute a two-dimensional refocused image, and the refocusing distance can be freely adjusted after the image is captured.

[0055] with dimensions [N x , N y (where N x and N y represent the number of pixels along the x-axis and y-axis, respectively) of the 4D light field image L is projected into a 2D image by shifting and scaling the microlens images, and then they are added together to form a 2D image. The shift amount of the microlens image controls the refocusing distance. The projection from the pixel at coordinates (x, y, i, j) in the 4D light field image L to the refocused 2D image coordinates (X, Y) is defined by the following equation:

[0056]

[0057] where s controls the size of the 2D refocused image and g controls the focusing distance of the refocused image. By considering Equation (1), Equation (6) can be rewritten as Equation (7):

[0058]

[0059] The parameter g can be expressed as a function of p and w as in Equation (8). The parameter g represents the scaling that must be performed on the microlens image, using the center of the microlens image as a reference, such that various scaled views of the same object are superimposed.

[0060]

[0061] Equation (7) becomes:

[0062]

[0063] Figure 4 is a schematic block diagram showing an example of an apparatus for obtaining a registration error map representing the level of blurriness of an image according to an embodiment of the present disclosure.

[0064] The apparatus 400 includes a processor 401, a storage unit 402, an input device 403, a display device 404, and an interface unit 405 connected via a bus 406. Of course, the constituent elements of the computer apparatus 400 may be connected by connections other than bus connections.

[0065] The processor 401 controls the operation of the device 400. The storage unit 402 stores at least one program to be executed by the processor 401, as well as various parameters, including data of the 4D light field image acquired and provided by the light field camera, parameters used in the calculations executed by the processor 401, intermediate data of the calculations executed by the processor 401, and so on. The processor 401 can be formed by any known and suitable hardware or software or a combination of hardware and software. For example, the processor 401 can be formed by dedicated hardware such as a processing circuit, or by a programmable processing unit such as a CPU (Central Processing Unit) that executes a program stored in its memory.

[0066] The storage unit 402 can be formed by any suitable memory or device capable of storing programs, data, etc. in a computer-readable manner. Examples of the storage unit 402 include non-transitory computer-readable storage media such as semiconductor memory devices, and magnetic, optical, or magneto-optical recording media loaded into a read and write unit. The program causes the processor 401 to execute the processing for obtaining a registration error map representing the blur level of an image as described below with reference to Figure 5 the processing for obtaining a registration error map representing the blur level of an image according to an embodiment of the present disclosure.

[0067] The input device 403 can be formed by a keyboard, a pointing device such as a mouse, etc. for use by a user to input commands, and the selection of a three-dimensional (or 3D) model of an object of interest by the user for defining a refocus plane. The output device 404 can be formed by a display device to display, for example, a graphical user interface (GUI), an image generated according to an embodiment of the present disclosure. For example, the input device 403 and the output device 404 can be integrally formed by a touch screen panel.

[0068] The interface unit 405 provides an interface between the device 400 and an external device. The interface unit 405 can communicate with the external device via wired or wireless communication. In an embodiment, the external device can be a light field camera. In this case, data of the 4D light field image acquired by the light field camera can be input from the light field camera to the device 400 through the interface unit 405 and then stored in the storage unit 402.

[0069] In this embodiment, it is exemplarily discussed that the device 400 is separated from the light field camera, and they can communicate with each other via wired or wireless communication. However, it should be noted that the device 400 can be integrated with such a light field camera. In the latter case, for example, the device 400 can be a portable device, such as a tablet computer or a smart phone embedded with a light field camera.

[0070] Figure 5 is a flowchart for explaining the processing for obtaining a registration error map representing the blur level of an image according to an embodiment of the present disclosure.

[0071] In the first stage Ph1 of the process for obtaining the error registration map, a refocusing plane is determined based on the three-dimensional model of the object of interest. The first stage Ph1 includes steps 501 to 503.

[0072] During step 501, the processor 401 executes a GUI function on the display 404 of the device 400. This GUI function allows the user of the device 400 to select a 3D model of the object of interest from among a plurality of 3D models of objects of interest stored in the storage unit 402 of the device 400. The user can select the 3D model on the GUI on the display 404 by pointing at the 3D model corresponding to the object of interest using a pointing device. In another embodiment of the present disclosure, the 3D model of the object of interest is automatically selected, for example, by decoding a multi-dimensional code associated with the object or scene. The multi-dimensional code is obtained, for example, by a camera embedded in the device 400, or is sent from an external device via wired or wireless communication. In the latter case, the decoding of the multi-dimensional code can be performed by the external device, and the result of the decoding is sent to the device 400. Then, the selected 3D model is stored in the storage unit 402.

[0073] Once the 3D model of the object of interest has been selected, the viewpoint of the object is selected in step 502. In an embodiment of the present disclosure, the viewpoint or reference position can be specified in a multi-dimensional code associated with the object of interest. In another embodiment of the present disclosure, the user of the device 400 can select the viewpoint himself, for example, using a pointing device, and position the 3D model at the position he has selected. Then, the information related to the selected viewpoint is stored in the storage unit 402 of the device 400.

[0074] In step 503, a distance map is calculated when the selected 3D model of the object of interest is set in the reference position. Referring to Figure 6 , the distance map 60 is a derived representation of the digital image 61. The distance map 60 labels each pixel 62 of the digital image 61 with the distance 63 to the nearest obstacle pixel. The most common type of obstacle pixel is the boundary pixel in the binary image 61. The distance map 60 is calculated by the processor 401 of the device 400.

[0075] Returning to Figure 5 , in the second stage Ph2 of the process for obtaining the error registration map, an image of the object of interest is calculated based on the 4D light field data related to the object of interest and the refocusing plane determined during the first stage Ph1, and a registration error map is obtained based on the refocused image. The second stage Ph2 includes steps 504 to 509.

[0076] In step 504, the apparatus 400 obtains 4D light field data related to an object of interest. In an embodiment of the present disclosure, the 4D light field data is obtained by an external device such as a light field camera. In this embodiment, the 4D light field data can be input into the apparatus 400 from the light field camera through the interface unit 405 and then stored in the storage unit 402. In another embodiment of the present disclosure, the apparatus 400 is embedded with a light field camera. In this case, the 4D light field data is obtained by the light field camera of the device 400 and then stored in the storage unit 402.

[0077] In step 505, the processor 401 calculates a focal stack based on the obtained 4D light field data related to the object of interest. Referring to Figure 7 , the focal stack 70 is a set of N refocused images R_n (where n ∈ [1, N]) that define an image cube, where N is the number of images selected by the user. Corresponding to the range of the focusing distance between z_min and z_max defined by equations (4) and (8), N refocused images are calculated for g linearly varying between g_min and g_max. Another option is to calculate the focal stack corresponding to the range of the focusing distance between z_min and z_max defined by equation (4), where w linearly varies from w_min to w_max. The minimum and maximum boundaries of g or w are defined by the user so as to include the refocused images with the focusing distance within z_min and z_max.

[0078] The calculation of the focal stack 70 described in this embodiment assumes that the 4D light field data is recorded by a single image sensor having a lens array and an optional main lens. However, the calculation of the focal stack 70 is not limited to the 4D light field data recorded by this type of light field camera, and thus it should be noted that a focal stack of refocused images can be calculated based on the 4D recorded by any type of light field camera.

[0079] Return Figure 5 , in step 506, the processor 401 calculates the intersection of the refocusing plane determined during the first stage Ph1 and the focal stack 70 calculated during step 505.

[0080] Referring to Figure 8 , for each pixel 80 of the coordinates (x, y) of the image to be refocused, the processor 401 determines the focusing distance z from the distance map calculated during step 503 during step 506. In fact, for each pixel (x, y) of the image, the corresponding distance map associates distance information Z. Thus, for each pixel (x, y) of the image 81 to be refocused, the focusing distance z is retrieved from the distance map by looking up the distance information Z associated with the pixel (x, y) of the distance map. The refocusing distance z corresponds to one of the images Rn that make up the focal stack 82.

[0081] Then returnFigure 5 In step 507, the processor 401 generates a refocused image by combining all pixels of the coordinates (x, y, z) that belong to the intersection of the focus stack and the refocus plane.

[0082] In step 608, the processor 401 of the apparatus 400 calculates a registration error map. As Figure 9 shown, the registration error map 90 labels each pixel 91 of the digital image 92 with information 93 related to the blur level of the pixel 92. For example, the information 93 related to blur can be a value between 0 and 1, where 0 represents the highest level of blur and 1 represents the highest level of sharpness (or the lowest level of blur), and the increment of the value indicating the blur level of the pixel is, for example, 0.25. In other words, the value indicating the blur level of the pixel can take the following values: 0, 0.25, 0.5, 0.75, or 1, where the value 0 indicates the highest value of blur and the value 1 indicates the lowest value of blur. Thus, Figure 9 the upper left corner of the image 92 shown is sharpened because the value of the information 93 representing the blur level associated with the pixel 91 in the upper left corner of the image 92 is equal to 1. In contrast, the lower right corner of the image 92 is blurred because the value of the information 93 representing the blur level associated with the pixel 91 in the lower right corner of the image 92 is equal to 0.

[0083] Return Figure 5 In step 509, the processor 401 triggers the display of the registration error map on the output device 404 of the apparatus 400. The registration error map displayed on the output device 404 can take on different appearances. In an embodiment of the present disclosure, the registration error map takes on the appearance of the refocused image itself; that is, the end user of the apparatus 400 sees the refocused image displayed on the output device 404, and some parts of the refocused image can appear blurred or sharpened depending on the value of the information indicating the blur level of the pixel.

[0084] In another embodiment of the present disclosure, the registration error map can take on the appearance of a two - color image. For example, if the value of the information related to blur is 0, the corresponding pixel appears red, and if the value of the information related to blur is 1, the corresponding pixel appears blue. For pixels having a value of the information related to blur between 0 and 1, the pixels appear in a mixed red shade or a mixed blue shade depending on their blur level. In this embodiment of the present disclosure, the processor 401 of the apparatus 400 determines the color associated with each pixel of the image based on the value of the information related to the blur level, and then generates the appearance of the registration error map to be displayed on the output device 404 of the apparatus 400.

[0085] The advantages of the method for obtaining a registration error map as described above are that it relies on using 4D light field data capable of generating images refocused on a composite refocusing surface such as a 3D surface. Once the composite refocusing surface is determined, a refocused image is obtained by calculating the intersection of the refocusing surface and the focal stack calculated from the obtained 4D light field data. For each pixel, refocused image information related to the blur level of the pixel is obtained, enabling the calculation of the registration error map. Since the refocusing surface is a complex surface, the registration error map reflects that the object of interest is not currently viewed at the correct viewing point, that is, the registration error map gives guidance information to the end user of the device 400 regarding how the end user should shift his viewing point around the initial point relative to the object of interest to obtain a sharp image of the object of interest.

[0086] Therefore, as long as the device 400 displays a sharp image of the object of interest on the output device 404, or once the registration error map indicates that the pixels corresponding to the object of interest are sharp, the end user knows that the object of interest is viewed at the correct viewing point.

[0087] This method provides a simple and user-friendly visual guidance solution.

[0088] An application of the method for obtaining a registration error map as described above is visual guidance where the end user is a human or a robot.

[0089] In an embodiment of the present disclosure, the end user is a person who owns a device 400 (such as a tablet computer) embedded with a light field camera.

[0090] Visual guidance is useful in situations where the end user must precisely position with respect to an object of interest (such as a statue in a museum), and at this time, the proper positioning of the statue triggers the display of information related to the statue on the output device 404 of the device 400.

[0091] First, a 3D model of the object of interest should be available. For example, such a 3D model of the object of interest may already be stored in the storage unit 402 of the device 400 because the end user, for example, downloaded an application developed by the museum that houses the statue. In another example, the 3D model of the object of interest is downloaded from a server and stored on the storage unit 402 after a trigger event occurs. Such a trigger event is, for example, the decoding of a multi-dimensional code related to the object of interest acquired by the camera of the device 400, and the multi-dimensional code embeds information from which the 3D model information of the object of interest can be downloaded, for example, a URL (Uniform Resource Locator).

[0092] Then, the first stage Ph1 of the method for obtaining a registration error map according to an embodiment of the present disclosure is executed by the processor 401 of the device 400. In an embodiment of the present disclosure, the first stage Ph1 of the method for obtaining a registration error map is executed by an external device. Then, the distance map obtained during the first stage Ph1 is input into the device 400 from the external device through the interface unit 405 and then stored in the storage unit 402.

[0093] When 4D light field data related to an object of interest is acquired, the second stage Ph2 of the method for obtaining a registration error map according to an embodiment of the present disclosure is executed, and the registration error map of the object of interest is displayed on the output device 404 of the device 400.

[0094] If the registration error map displayed on the output device 404 of the device 400 appears as a blurred image, it means that the end user of the device 400 has not correctly positioned the object of interest, that is, the object of interest is not viewed according to the viewpoint that determines the refocusing plane.

[0095] Based on this visual feedback, the end user shifts his viewpoint until the currently displayed registration error map on the output device 404 of the device 400 is sharpened, which means that the appropriate object of interest is viewed at the appropriate viewpoint, that is, viewed according to the viewpoint that determines the refocusing plane.

[0096] The intensity and shape of the blurriness of the registration error map displayed on the output device 404, or the amount of the color red or blue indicating the blurriness level, helps the end user determine in which direction he should move the device 400 to shift the currently viewed viewpoint of the object of interest.

[0097] When the end user is a robot, the processor 401 can send information related to the blurriness level of the pixels of the digital image to the device controlling the movement of the robot through the interface unit 405 of the device 400.

[0098] Based on this information related to the blurriness of the pixels of the digital image, the device controlling the movement of the robot can determine in which direction the robot must move to obtain a sharpened image of the object of interest.

[0099] In addition, a positioning system such as GPS (Global Positioning System) or any indoor positioning system can be used to help the end user find the appropriate position to view the object of interest.

[0100] In an embodiment of the present invention, in order to increase the visual guidance effect, the blurriness of the registration error map can be intentionally exaggerated in some areas of the registration error map. Therefore, based on this visual feedback, the end user knows in which direction to move the device 400 to reach the reference position.

[0101] Another application of the method for obtaining a registration error map as described above is to inspect a surface during a quality control process performed by an end user who is either a human or a robot.

[0102] In an embodiment of the present disclosure, the end user is a human who owns a device 400 (such as a tablet computer) with an embedded light field camera.

[0103] In a case where the end user has to inspect the surface of an object of interest (such as a part of an aircraft wing), it may be useful to refocus the image on a refocus plane.

[0104] A 3D model of the part of the aircraft wing of interest should be available. For example, such a 3D model of the part of the aircraft wing to be inspected may already be stored in the storage unit 402 of the device 400.

[0105] Then, a first stage Ph1 of the method for obtaining a registration error map according to an embodiment of the present disclosure is executed by the processor 401 of the device 400. In an embodiment of the present disclosure, the first stage Ph1 of the method for generating a registration error map is executed by an external device. Then, the distance map obtained in the first stage Ph1 is input into the device 400 from the external device through the interface unit 405 and then stored in the storage unit 402.

[0106] When acquiring 4D light field data related to the part of the aircraft wing to be inspected, a second stage Ph2 of the method for generating a registration error map according to an embodiment of the present disclosure is executed, and the registration error map of the part of the aircraft wing to be inspected is displayed on the output device 404 of the device 400.

[0107] If the registration error map displayed on the output device 404 of the device 400 appears as a blurred image, it means that the surface of the aircraft wing has a defect. In fact, when the surface of the part of the aircraft wing being inspected is defective compared to the 3D model of the defect-free refocus plane, the defect of the surface of the part of the aircraft wing being inspected is indicated as blurring on the registration error map displayed on the output device 404 of the device 400.

[0108] In this embodiment, it is assumed that the end user is viewing the part of the aircraft wing being inspected from an appropriate viewing point, so the blurriness represented by the registration error map is due to the defect of the surface of the part of the aircraft wing. For example, the end user can use a positioning system such as GPS or any indoor positioning system to find an appropriate position to view the object of interest.

[0109] In an embodiment of the present disclosure, in order to make it easier for the end user to detect the defect, the blurriness indicated by the registration error map can be intentionally exaggerated in certain regions of the registration error map.

[0110] Although the present disclosure has been described above with reference to specific embodiments, the present disclosure is not limited to the specific embodiments, and modifications falling within the scope of the present disclosure will be apparent to those skilled in the art.

[0111] After referring to the above illustrative embodiments, many further modifications and variations will be made by those skilled in the art, which are given by way of example only and are not intended to limit the scope of the present disclosure, which is determined only by the appended claims. Specifically, different features from different embodiments may be interchanged where appropriate.

Claims

1. A method for generating a refocused image, comprising: Obtaining a 3D model of an object of interest associated with a scene; Determining a refocus plane by calculating a distance map of the 3D model when the object of interest is set in a reference position, the distance map marking pixels of a digital image with the distance to the nearest obstacle pixel; Obtaining four-dimensional light field data of the scene; Refocusing the four-dimensional light field data using the refocus plane by determining a focusing distance from the distance map; Generating the refocused image by using the refocused four-dimensional light field data and the focusing distance; And Determining whether the end user is correctly positioned relative to the scene by using the refocused image.

2. The method according to claim 1, wherein The 3D model of the object of interest is obtained according to the selection of the end user.

3. The method according to claim 1, wherein The 3D model of the object of interest is obtained according to the selection of a code identifying the scene.

4. The method according to claim 1, wherein The end user is a human user.

5. The method according to claim 4, wherein Displaying the refocused image on a mobile device such that the human user views the refocused image.

6. The method according to claim 1, wherein, The end user is a robot.

7. The method according to claim 1, wherein The four-dimensional light field data of the scene is obtained from the current viewpoint of the end user by using a camera device.

8. The method according to claim 1, wherein, Determining whether the end user is correctly positioned relative to the scene by using the refocused image includes: Generating a registration error map representing the sharpness levels of multiple pixels of the image.

Images