Information processing device and information processing method

Abstract

An information processing device (200A, 200B, 200C) according to the present invention is provided with a control unit (220, 220B, 220C). The control unit (220, 220B, 220C) acquires an image of an object captured by a sensor. The captured image is an image that is obtained from reflection of light radiated on the object from a plurality of light sources arranged at different positions. The control unit (220, 220B, 220C) extracts a flat area from the captured image on the basis of luminance values in the captured image. The control unit (220, 220B, 220C) calculates shape information about the surface shape of the object on the basis of the flat area of the captured image and information about the sensor.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] Microscopes are widely used as devices for observing the minute conditions of objects because they are relatively inexpensive to introduce and can perform measurements easily.

[0003] As a technology related to a microscope, a technology is known that uses the difference in the incidence angle of the lighting unit to analyze the color and dirt on the skin surface (for example, Patent Document 1). In addition, a technology is known that reduces out-of-focus and distortion when photographing the skin surface by placing transparent glass at a specified distance from the dome at the tip of the microscope (for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 10-333057 [Patent Document 2] JP 2008-253498 A Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional techniques can improve the quality of images captured by a microscope.

[0006] However, conventional technologies are only intended to improve the quality of planar images, and it is difficult to obtain 3D images that reproduce the minute shapes (convex and concave) of objects. Non-contact 3D measuring devices and 3D scanners are used to measure the minute shapes of objects, but the introduction of these devices entails a problem of relatively high costs. Distance measuring devices using the ToF (Time of Flight) method are relatively inexpensive, but may not be sufficiently accurate.

[0007] Therefore, the present disclosure provides an information processing device and an information processing method that can improve the accuracy of shape measurement. [Means for solving the problem]

[0008] According to the present disclosure, an information processing device is provided. The information processing device includes a control unit. The control unit acquires an image of an object captured by a sensor. The captured image is an image obtained from reflected light of light irradiated onto the object from a plurality of light sources arranged at different positions. The control unit extracts a flat region from the captured image based on a luminance value of the captured image. The control unit calculates shape information relating to a shape of a surface of the object based on information related to the sensor and the flat region of the captured image. [Brief description of the drawings]

[0009] [Figure 1] 11A and 11B are diagrams for explaining a method for calculating a depth from a captured image. [Diagram 2] 11 is a table for explaining a rough shape of a target surface. [Diagram 3] FIG. 1 is a diagram for explaining an overview of a first embodiment of the present disclosure. [Figure 4] 1 is a block diagram showing a configuration example of an information processing system according to a first embodiment of the present disclosure. [Diagram 5] FIG. 1 is a diagram illustrating an example of the configuration of a microscope according to a first embodiment of the present disclosure. [Figure 6]1 is a block diagram showing a configuration example of an information processing device according to a first embodiment of the present disclosure. [Figure 7] FIG. 4 is a diagram illustrating an example of luminance values ​​of a captured image. [Figure 8] FIG. 13 is a diagram showing an example of a mask image generated by a region acquisition unit. [Figure 9] 11A and 11B are diagrams for explaining an example of acquiring a flat region by a region acquiring unit; [Figure 10] FIG. 11 is a diagram illustrating an example of normal information. [Figure 11] FIG. 13 is a diagram for explaining a learning device according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a diagram for explaining the length per pixel of a captured image. [Figure 13] 4 is a diagram showing an example of an image that a display control unit causes to be displayed on a display unit; FIG. [Figure 14] 11 is a flowchart for explaining an example of a depth calculation process according to the first embodiment of the present disclosure. [Figure 15] 13 is a diagram for explaining a captured image acquired by an acquisition section according to a second embodiment of the present disclosure. FIG. [Figure 16] 13 is a diagram for explaining a captured image acquired by an acquisition section according to a second embodiment of the present disclosure. FIG. [Figure 17] 13A to 13C are diagrams for explaining a smoothing process performed by a region acquisition section according to the third embodiment of the present disclosure. [Figure 18] 13A to 13C are diagrams for explaining acquisition of a flat region by a region acquisition section according to a fourth embodiment of the present disclosure. [Figure 19] 13A to 13C are diagrams for explaining acquisition of a flat region by a region acquisition section according to a fourth embodiment of the present disclosure. [Figure 20] FIG. 2 is a diagram for explaining the depth of field of a sensor. [Figure 21] 13A to 13C are diagrams for explaining acquisition of a flat region by a region acquisition section according to a fifth embodiment of the present disclosure. [Figure 22]13A to 13C are diagrams for explaining acquisition of a flat region by a region acquisition section according to a fifth embodiment of the present disclosure. [Figure 23] 13A to 13C are diagrams for explaining a plurality of captured images acquired by an acquisition section according to a sixth embodiment of the present disclosure. [Figure 24] 23A to 23D are diagrams for explaining acquisition of a flat region by a region acquisition section according to a sixth embodiment of the present disclosure. [Diagram 25] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a seventh embodiment of the present disclosure. [Figure 26] FIG. 23 is a diagram for explaining frequency separation by a normal frequency separation section according to a seventh embodiment of the present disclosure. [Figure 27] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to an eighth embodiment of the present disclosure. [Figure 28] FIG. 23 is a diagram for explaining frequency separation by a normal frequency separation section according to an eighth embodiment of the present disclosure. [Figure 29] FIG. 13 is a diagram for explaining a learning method of a learning module according to the ninth embodiment. [Diagram 30] 23 is a diagram illustrating an example configuration of a control unit of an information processing device according to a tenth embodiment of the present disclosure. FIG. [Diagram 31] 1 is a block diagram showing an example of a schematic configuration of a patient's in-vivo information acquisition system using a capsule endoscope to which the technology according to the present disclosure (the present technology) can be applied. [Diagram 32] 1 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied. [Diagram 33] 33 is a block diagram showing an example of a functional configuration of the camera head and the CCU shown in FIG. 32. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] The explanation will be given in the following order. 1. Background 1.1. How to calculate surface shape 1.2. Issues with calculation methods 2. First embodiment 2.1. Overview of the first embodiment 2.2. System configuration example 2.3.Microscope configuration example 2.4. Example of information processing device configuration 2.5.Depth Calculation Process 3. Second embodiment 4. Third embodiment 5. Fourth embodiment 6. Fifth embodiment 7. Sixth embodiment 8. Seventh embodiment 9. Eighth embodiment 10. Ninth embodiment 11. Tenth embodiment 12. Other embodiments 13.Application Examples 14. Supplementary Information

[0012] <1. Background> <1.1. Surface shape calculation method> First, before describing the details of the embodiments of the present disclosure, the background that led the inventors to create the embodiments of the present disclosure will be described.

[0013] As a method for obtaining a 3D image that reproduces the shape (convex and concave) of an object from an image (RGB image) captured by an imaging device, a method is known in which the surface shape (depth to the surface) is calculated directly from the RGB image using machine learning such as CNN (Convolutional Neural Network).However, when calculating the depth directly from the RGB image using CNN, there is uncertainty, making it difficult to improve the accuracy.

[0014] Another method for calculating the depth from a captured image to an object surface is to calculate normal information of the object surface from the captured image using machine learning such as CNN, and convert the normal information into depth. This method will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining a calculation method for calculating depth from a captured image. The method described with reference to Fig. 1 is executed by, for example, an information processing device 200.

[0015] In this method, the information processing device 200 first acquires a captured image captured by a microscope (step S1). At this time, the information processing device 200 also acquires information about the microscope, for example. The information processing device 200 calculates a normal (normal information) on the surface of the object based on the captured image (step S2). The information processing device 200 obtains normal information as output data by inputting the captured image to a learning device (model) trained using, for example, CNN.

[0016] The information processing device 200 calculates the distance (depth information) to the surface of the object based on the normal information (step S3). If the distance between the object and the sensor of the microscope, which is one of the pieces of information about the microscope, is known, the information processing device 200 can measure the distance to the surface of the object based on the normal information. Here, when a user images an object with a microscope, the image is captured by bringing the head mount of the microscope into contact with the object. Therefore, the distance between the sensor of the microscope and the object is a known value that corresponds to the length of the head mount.

[0017] Specifically, the information processing device 200 calculates the depth to the object surface based on the normal information by minimizing W in the following formula (1).

[0018]

number

[0019] The parameters in equation (1) are as follows: p: x direction of the calculated normal q: y direction of the calculated normal Z x : Partial differential of the desired depth in the x direction (x direction of the normal) Z y : Partial derivative of the desired depth in the y direction (y direction of the normal) Z xx : The second partial derivative of the desired depth in the x direction (the partial derivative of the normal in the x direction) Z yy : The second partial derivative of the desired depth in the y direction (the partial derivative of the normal in the y direction) Z xy : Second partial differential of the desired depth in the x and y directions

[0020] Note that x and y indicate coordinates on the captured image, with the x direction being, for example, the horizontal direction of the captured image and the y direction being the vertical direction of the captured image.

[0021] The Fourier transform (frequency transform) of equation (1) gives equation (2). By minimizing this, the depth to the target surface is calculated. Note that Z in equation (2) F is the Fourier transform of the desired depth, and u, v represent coordinates in frequency space.

[0022]

number

[0023] Equation (2) is expanded into the following equation (3), and the depth to the target surface is calculated by performing an inverse Fourier transform on equation (3).

[0024]

number

[0025] Using the above-mentioned formula conversion of formulas (1) to (3), the information processing device 200 calculates the distance (depth) to the surface of the object.

[0026] <1.2. Issues with calculation methods> In the above-mentioned calculation method, the second and third lines on the right-hand side of equation (1) are cost terms in equation (1), with the second line representing the sum of the absolute values ​​of the normals and the third line representing the sum of the absolute values ​​of the normal derivatives.

[0027] Here, in the above-mentioned calculation method, the depth is calculated under the assumption that the weights (λ, μ) of the cost terms are small, i.e., the sum of the absolute values ​​of the normals is small and the sum of the absolute values ​​of the normal derivatives is small. The assumption that the sum of the absolute values ​​of the normals is small (hereinafter also referred to as assumption 1) means that the target surface for which the depth is to be calculated is flat. In addition, the assumption that the sum of the absolute values ​​of the normal derivatives is small (hereinafter also referred to as assumption 2) means that the curvature of the target surface for which the depth is to be calculated is small. In other words, in the above-mentioned calculation method, the depth to the target is calculated under the assumption that the rough surface shape of the target for which the depth is to be calculated is flat.

[0028] Here, the rough shape of the target surface will be explained using Fig. 2. Fig. 2 is a table for explaining the rough shape of the target surface. In Fig. 2, the actual shape of the target surface is shown by a dotted line, and the rough shape of the target surface is shown by a straight line. Also, the normal to the target surface is shown by an arrow. In Fig. 2, the horizontal direction of the target is the x direction, and the vertical direction is the z direction (depth).

[0029] The rough shape of the target surface is the shape of the target surface in the entire captured image, and the rough shape being flat means that there is little variation when looking at the shape of the target surface in the entire captured image. Note that local unevenness changes on the target surface are not included in the rough shape because they are subject to calculation. The more the assumption of the cost term in the optimization formula shown in the above formula (1) is satisfied, that is, the closer the surface shape of the target surface in the entire captured image is to a flat surface, the more accurately the information processing device 200 can calculate the depth to the target surface.

[0030] For example, the object shown in (1) in the table of Fig. 2 has a shape including fine irregularities on a straight surface. Therefore, the object shown in (1) satisfies both of the above-mentioned assumptions 1 and 2 (indicated by "○" in the table), and the information processing device 200 can calculate the depth of the object with high accuracy.

[0031] The object shown in (2) in the table of Fig. 2 has a shape that includes fine irregularities on a surface that is slightly curved upward. Therefore, the object shown in (2) somewhat satisfies the above-mentioned assumption 1 (indicated by "△" in the table) and satisfies assumption 2. Therefore, the information processing device 200 can calculate the depth of the object shown in (2) with lower accuracy than that of (1), but with good accuracy.

[0032] The object shown in (3) in the table of Fig. 2 has a shape that includes fine irregularities on the surface that curves significantly upward on the left side. Therefore, the object shown in (3) does not satisfy either of the above-mentioned assumptions 1 and 2 (indicated by "x" in the table). Therefore, the information processing device 200 cannot accurately calculate the depth of the object shown in (3).

[0033] The object shown in (4) in the table of Fig. 2 has a stepped surface. As described above, the object shown in (4) has a shape including fine irregularities on a surface having multiple straight faces with different heights and directions, and therefore does not satisfy either of the above-mentioned assumptions 1 and 2. Therefore, the information processing device 200 cannot accurately calculate the depth of the object shown in (4).

[0034] As described above, the calculation method described above has a problem in that the accuracy of the depth calculation decreases if the assumption that the rough shape of the surface of the object for which the depth is to be calculated is flat is not satisfied.

[0035] In view of such circumstances, the present disclosure has come to create each embodiment of the present disclosure related to the information processing device 200 that can improve the accuracy of shape measurement by improving the accuracy of depth calculation. Hereinafter, the details of each embodiment of the present disclosure will be described in order.

[0036] 2. First embodiment <2.1. Overview of the first embodiment> 3 is a diagram for explaining an overview of the first embodiment of the present disclosure. An information processing device 200A (not shown) according to the first embodiment extracts an area (flat area) that satisfies the above-mentioned assumptions 1 and 2 from a captured image, thereby suppressing deterioration in accuracy of shape information (depth information) related to the surface shape of an object and improving accuracy of shape measurement.

[0037] The information processing device 200A first acquires a captured image M11 of an object S captured by the sensor 150 of the microscope 100. The captured image M11 is an image obtained from reflected light of light IA and IB irradiated onto the object S from a plurality of light sources 160A and 160B disposed at different positions.

[0038] Here, the microscope 100 will be briefly described. As shown in the left diagram of Fig. 3, the microscope 100 has a sensor 150, a head mount unit 10 which is a cylindrical mechanism installed between the sensor 150 and an imaging target, and a plurality of light sources 160A, 160B. The sensor 150 may be interpreted as a lens, a camera, or the like. The microscope 100 is an imaging device which is held by a user and used by directing the sensor 150 to the target S by bringing the head mount unit 10 into contact with the target S.

[0039] The microscope 100 exposes the target S to the reflected light of lights IA and IB that are simultaneously irradiated from light sources 160A and 160B, and captures an image of the target S. At this time, for example, if the rough shape of the surface of the target S is not flat, an occlusion (a region where light does not hit) occurs on the surface of the target S.

[0040] For example, light IA emitted from light source 160A hits areas SA and SAB on the surface of object S, but does not hit area SB. On the other hand, light IB emitted from light source 160B hits areas SB and SAB on the surface of object S, but does not hit area SA. Area SAB, where light IA and IB from multiple light sources 160A and 160B both hit, is a flat area with no occlusion.

[0041] As shown in the right diagram of FIG. 3, in the captured image M11 of the microscope 100, the areas SA and SB that are hit by only one of the lights IA and IB from the light sources 160A and 160B are darker and have a lower brightness value than the area SAB that is hit by both lights IA and IB.

[0042] Therefore, the information processing device 200A extracts a flat region SAB from the captured image M11 based on the luminance value of the captured image M11. For example, the information processing device 200A extracts a region SAB in which the luminance value of the captured image M11 is equal to or greater than a threshold value as a flat region.

[0043] The information processing device 200A calculates shape information (depth information) regarding the shape of the surface of the target S based on information regarding the sensor 150 and the flat region SAB of the captured image M11. For example, the information processing device 200A acquires normal information of the flat region SAB by inputting the flat region SAB of the captured image M11 to a learning device trained using CNN. The information processing device 200A calculates depth information to the surface of the target S by performing equation conversion on the acquired normal information using the above-mentioned equations (1) to (3).

[0044] In this way, the information processing device 200A can improve the accuracy of shape measurement by extracting an area that satisfies assumptions 1 and 2 of formula (1) from the captured image M11.

[0045] Hereinafter, the information processing system 1 having the above-mentioned information processing device 200A will be described in detail.

[0046] <2.2. System configuration example> Fig. 4 is a block diagram showing an example of the configuration of the information processing system 1 according to the first embodiment of the present disclosure. As shown in Fig. 4, the information processing system 1 includes a microscope 100 and an information processing device 200A.

[0047] The microscope 100 is an imaging device that is held by a user and used by pointing the sensor 150 toward an imaging target S.

[0048] The information processing device 200A calculates shape information relating to the surface shape of the imaging target S, based on the captured image M11 captured by the microscope 100. Details of the information processing device 200A will be described later with reference to FIG.

[0049] The microscope 100 and the information processing device 200A are connected, for example, using a cable. Alternatively, the microscope 100 and the information processing device 200A may be directly connected by wireless communication such as Bluetooth (registered trademark) or NFC (Near Field Communication). The microscope 100 and the information processing device 200A may be connected, for example, via a network (not shown), by wire or wirelessly. Alternatively, the microscope 100 and the information processing device 200A may exchange captured images via an externally mounted storage medium such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card. The microscope 100 and the information processing device 200A may be configured as an integrated unit. Specifically, the information processing device 200A may be configured to be disposed, for example, inside the main body of the microscope 100.

[0050] <2.3. Microscope configuration example> 5 is a diagram showing a configuration example of a microscope 100 according to the first embodiment of the present disclosure. The microscope 100 has a sensor 150 and a head mount unit 10, which is a cylindrical mechanism installed between the sensor 150 and an imaging target. The microscope 100 also has point light sources 160A and 160B (not shown) that are arranged at different positions.

[0051] The head mount unit 10 is a mechanism mounted on the tip of the microscope 100. The head mount unit 10 is also called, for example, a tip head or a lens barrel. The inside of the head mount unit 10 may be, for example, a mirror, so that light irradiated from point light sources 160A and 160B is totally reflected by the side surface of the head mount unit 10.

[0052] The user brings the head mounted unit 10 into contact with the target S to capture an image of the target S. This fixes the distance between the sensor 150 and the target S, making it possible to prevent the focus (focal length) from shifting during imaging.

[0053] The microscope 100 has point light sources 160A and 160B inside the head mount unit 10. As a result, the microscope 100 exposes the target S to the reflected light of light IA and IB irradiated from the point light sources 160A and 160B, and captures an image of the target S. Note that the point light source in this specification ideally means a light source based on a point, but since a point light source does not actually exist, it includes a light source with an extremely small size (within a few millimeters, for example).

[0054] In this specification, a case where a point light source is used as an example of a light source will be described, but the light source is not limited to a point light source. The number of light sources is also not limited to two. There may be more than one light source, and there may be three or more light sources as long as they are arranged at different positions.

[0055] <2.4. Example of configuration of information processing device> 6 is a block diagram showing an example of a configuration of an information processing device 200A according to the first embodiment of the present disclosure. The information processing device 200A includes a control unit 220 and a storage unit 230.

[0056] (Control unit) The control unit 220 controls the operation of the information processing device 200A. The control unit 220 has an acquisition unit 221, a region acquisition unit 225, a normal calculation unit 222, a depth calculation unit 223, and a display control unit 224. The acquisition unit 221, the region acquisition unit 225, the normal calculation unit 222, the depth calculation unit 223, and the display control unit 224 are realized by, for example, the control unit 220 executing a program stored in the control unit 220 using a RAM or the like as a working area. The internal structure of the control unit 220 is not limited to the configuration shown in FIG. 6, and may be other configurations as long as it is configured to perform information processing described later. Furthermore, the connection relationship of each processing unit included in the control unit 220 is not limited to the connection relationship shown in FIG. 6, and may be other connection relationships.

[0057] (Acquisition Department) The acquisition unit 221 acquires a captured image M11 captured by the microscope 100 and information about the microscope 100. The information about the microscope 100 includes information about the structure of the microscope 100, such as the focal length f and the length d of the head mount unit 10.

[0058] The acquisition unit 221 may control, for example, the point light sources 160A and 160B and the sensor 150 of the microscope 100. In this case, the acquisition unit 221 controls the point light sources 160A and 160B so that the light sources 160A and 160B simultaneously irradiate the light IA and IB. The acquisition unit 221 also controls the sensor 150 so that the sensor 150 images the target S while the light sources 160A and 160B simultaneously irradiate the light IA and IB. In this manner, the information processing device 200A may control the microscope 100.

[0059] Alternatively, the acquisition unit 221 may acquire information on the imaging conditions in addition to the captured image M11. The information on the imaging conditions is, for example, information indicating that the captured image M11 is an image captured while the lights IA and IB are simultaneously irradiated from the point light sources 160A and 160B. In this case, the region acquisition unit 225 described later extracts a flat region SAB from the captured image M11 according to, for example, the imaging conditions.

[0060] (Area acquisition part) The region acquisition unit 225 extracts a flat region SAB from the captured image M11. For example, the region acquisition unit 225 compares the luminance value L(x, y) of each pixel of the captured image M11 with a threshold th. As shown in FIG. 7, the region acquisition unit 225 sets a region including pixels whose luminance value L(x, y) is equal to or greater than the threshold th as a processing region, and sets a region including pixels whose luminance value L(x, y) is less than the threshold th as an excluded region. Note that FIG. 7 is a diagram showing an example of the luminance value of the captured image M11, and shows the relationship between the luminance value L(x, β) and pixel (x, β) when the value of the y-axis is a predetermined value "β".

[0061] If the captured image M11 is an RGB image, the luminance value L(x, y) is calculated as L(x, y) = (R(x, y) + 2G(x, y) + B(x, y)) / 4, where R(x, y) is the R (Red) component of the pixel value at pixel (x, y), G(x, y) is the G (Green) component of the pixel value at pixel (x, y), and B(x, y) is the B (Blue) component of the pixel value at pixel (x, y).

[0062] The region acquisition unit 225 compares the luminance value L(x, y) of all pixels with the threshold th, and acquires the region determined to be the processing region as an extraction region (flat region SAB) to be extracted from the captured image M11.

[0063] The region acquiring unit 225 generates, for example, a mask image M12 shown in Fig. 8 by coloring a pixel white when its luminance value L(x, y) is equal to or greater than a threshold th, and coloring a pixel black when its luminance value L(x, y) is less than the threshold th. Note that Fig. 8 is a diagram showing an example of the mask image M12 generated by the region acquiring unit 225.

[0064] The region acquiring unit 225 acquires the flat region SAB by comparing the generated mask image M12 with the captured image M11 and extracting pixels in which the mask image M12 is white at the same coordinates from the captured image M11, as shown in Fig. 9. Note that Fig. 9 is a diagram for explaining an example of acquiring the flat region SAB by the region acquiring unit 225.

[0065] Alternatively, the region acquiring unit 255 may generate the mask image M12 by setting the luminance value of a pixel to "1" when the luminance value L(x, y) of the pixel is equal to or greater than the threshold th, and setting the luminance value of the pixel to "0" when the luminance value L(x, y) is less than the threshold th. In this case, the region acquiring unit 255 acquires the flat region SAB by multiplying the captured image M11 and the mask image M12.

[0066] The area acquisition unit 255 outputs the acquired flat area SAB to the normal line calculation unit 222. Note that the area acquisition unit 255 may output the generated mask image M12 to the normal line calculation unit 222, and the normal line calculation unit 222 may acquire the flat area SAB from the captured image M11 using the mask image M12.

[0067] (Normal calculation part) The normal calculation unit 222 calculates normal information on the surface of the target S as a normal image based on the flat region SAB of the captured image M11. The normal calculation unit 222 generates the normal image using, for example, CNN. The normal calculation unit 222 generates the normal image using a learning device that has been trained in advance using CNN. The number of input channels of the learning device 300 (see FIG. 11) is three channels corresponding to the RGB of the input image (flat region SAB in this case) or one channel of the grayscale of the input image. The number of output channels of the learning device 300 is three channels corresponding to the RGB of the output image (normal image in this case). The resolution of the output image is equal to the resolution of the input image.

[0068] Here, the relationship between the normal information and RGB of the normal image will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining an example of normal information. The normal N shown in FIG. 10 is assumed to be a normal vector at a predetermined pixel of an image M11 obtained by capturing an image of an object S. In this case, if the zenith angle of the normal N is θ and the azimuth angle is φ, the relationship of the following formula (4) holds between the normal N(θ, φ) in the polar coordinate system and the normal N(x, y, z) in the Cartesian coordinate system. Note that the normal N is assumed to be a unit vector.

[0069]

number

[0070] The normal information is the normal N(x, y, z) in the above-mentioned Cartesian coordinate system, and is calculated for each pixel of the flat area SAB of the captured image M11, for example. The normal image is an image obtained by replacing the normal information of each pixel with RGB. In other words, the normal image is an image obtained by replacing the x of the normal N with the value of R (red), y with the value of G (green), and z with the value of B (blue).

[0071] Returning to Fig. 6, the normal calculation unit 222 generates a normal image, which is an output image, by inputting the flat area SAB as an input image to the learning device 300. The learning device 300 is assumed to have generated a normal image in advance by learning the weights in the CNN using the true learning value as the normal.

[0072] Here, a method for generating the learning device 300 will be described with reference to Fig. 11. Fig. 11 is a diagram for describing the learning device 300 according to an embodiment of the present disclosure. Here, for the sake of simplicity, the description will be given assuming that the control unit 220 of the information processing device 200A generates the learning device 300, but the learning device 300 may be generated by, for example, another information processing device (not shown).

[0073] In the example shown in FIG. 11, the information processing device 200A calculates the similarity between an output image M15 when an input image M14 is input to the learning device 300, and a normal image M16 which is the ground truth of learning. Here, the information processing device 200A calculates the least square error (L2) between the output image M15 and the normal image M16, and performs a learning process using L2 as a loss. The information processing device 200A updates the weight of the learning device 300, for example, by backpropagating the calculated loss. In this way, the information processing device 200A generates a learning device 300 which outputs a normal image when a captured image is input.

[0074] In this embodiment, the learning device 300 is generated using CNN as an example of machine learning, but is not limited to this. The learning device 300 may be generated using various methods other than CNN, such as RNN (Recurrent Neural Network), as machine learning. In the above example, the weight of the learning device 300 is updated by backpropagating the calculated loss, but is not limited to this. The weight of the learning device 300 may be updated using any learning method other than backpropagation, such as stochastic gradient descent. In the above example, the loss is the least square error, but is not limited to this. The loss may be the minimum mean error.

[0075] Returning to Fig. 6, the normal calculation unit 222 generates a normal image by using the generated learning device 300. The normal calculation unit 222 outputs the calculated normal image to the depth calculation unit 223.

[0076] (Depth calculation part) The depth calculation unit 223 receives each pixel value of the normal image as input, and performs equation conversion using the above-mentioned equations (1) to (3) to calculate the distance (depth) from the sensor 150 to the target S. Specifically, the depth calculation unit 223 performs an inverse Fourier transform on the above-mentioned equation (3) to calculate Z', and calculates the desired depth Z as Z=Z'×P, as follows.

[0077]

number

[0078] The parameters in the above-mentioned equation (3) are as follows: P(u, v): Fourier transform of the normal in the x direction Q(u, v): Fourier transform of the normal in the y direction u, v: Coordinates of each pixel in frequency space P: Length per pixel of the captured image xy [μm] Z F :Fourier transform of the desired depth Z': Desired depth per pixel [pixel] Z: Desired depth [μm]

[0079] The above P (length [μm] per pixel of the captured image xy) is a parameter whose value is determined by the configuration of the sensor 150 of the imaging device (microscope 100).

[0080] For example, as shown in Fig. 12, the focal length of the sensor 150 is f [mm], and the length of the housing (head mount unit 10) is d [mm]. If the pixel pitch of the sensor 150 is p [μm], the actual length P of one pixel of the captured image is P = (d × p) / f [μm], since P: d = p: f. Note that Fig. 12 is a diagram for explaining the length per pixel of the captured image.

[0081] The depth calculation unit 223 calculates the depth of the target S for each pixel of the captured image M11, for example, based on the normal image and information about the sensor 150. The information about the sensor 150 is, for example, information about the structure of the sensor 150, and specifically, information about the focal length of the sensor 150 and the distance between the sensor 150 and the target S. When the imaging device is the microscope 100, the distance between the sensor 150 and the target S is the length d of the head mount unit 10.

[0082] (Display control unit) The display control unit 224 displays various images on a display unit (not shown) such as a liquid crystal display. Fig. 13 is a diagram showing an example of an image M17 displayed on the display unit by the display control unit 224. As shown in Fig. 13, the display control unit 224 displays an image M17 including, for example, a captured image M17a, a normal image M17b, a depth image M17c, an image M17d showing a depth graph, and a 3D image M17e of the target S on the display unit.

[0083] The captured image M17a is, for example, an image obtained by cutting out the flat area SAB from the captured image M11 captured by the microscope 100. The normal image M17b is an image in which normal information of the captured image M17a is displayed in RGB. The depth image M17c is an image showing the depth of each pixel of the captured image M17a, and for example, the lighter the color, the greater the depth (the greater the distance from the sensor 150 to the target S). The image M17d showing the depth graph is an image in which the depth of a straight line shown in the normal image M17b, the depth image M17c, and the 3D image M17e of the target S is displayed as a graph. The 3D image M17e of the target S is an image in which the target S is displayed three-dimensionally based on the depth image M17c.

[0084] In this manner, the display control unit 224 may cause the display unit to display a graph indicating depth and a three-dimensional image in addition to the captured image and the generated normal image and depth image.

[0085] (Storage part) The storage unit 230 is realized by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 220, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0086] <2.5. Depth calculation process> Fig. 14 is a flowchart for explaining an example of a depth calculation process according to the first embodiment of the present disclosure. The depth calculation process shown in Fig. 14 is realized by the control unit 220 of the information processing device 200A executing a program. The depth calculation process shown in Fig. 14 is executed after the microscope 100 captures the captured image M11. Alternatively, the depth calculation process shown in Fig. 14 may be executed in response to an instruction from a user.

[0087] 14, the control unit 220 acquires the captured image M11 from, for example, the microscope 100 (step S101). Alternatively, the control unit 220 may acquire the captured image M11 from another device via, for example, a network (not shown). Examples of the other device include another information processing device and a cloud server.

[0088] The control unit 220 acquires information about the sensor 150 (step S102). The control unit 220 may acquire information about the sensor 150 from the microscope 100. Alternatively, when sensor information about the sensor 150 is stored in the storage unit 230, the control unit 220 may acquire the sensor information from the storage unit 230. Furthermore, the control unit 220 may acquire the sensor information by, for example, an input from a user.

[0089] The control unit 220 acquires a flat region SAB from the acquired captured image M11 (step S103). Specifically, the control unit 220 extracts a region in which the luminance value of each pixel of the captured image M11 is equal to or greater than a threshold value th as the flat region SAB.

[0090] The control unit 220 calculates normal information of the acquired flat region SAB (step S104). Specifically, the control unit 220 inputs the flat region SAB as an input image to the learning device 300 to generate a normal image including the normal information as an output image.

[0091] The control unit 220 calculates depth information based on the normal information and information related to the sensor 150 (step S105). Specifically, the control unit 220 calculates the depth information by performing equation conversion on the normal information based on the above-mentioned equation (3) or the like.

[0092] As described above, the information processing device 200A according to the first embodiment includes the control unit 220. The control unit 220 acquires a captured image M11 of the target S captured by the sensor 150. The captured image M11 is an image obtained from reflected light of light IA and IB irradiated onto the target S from a plurality of point light sources 160A and 160B disposed at different positions. The control unit 220 extracts a flat area SAB from the captured image M11 based on the luminance value of the captured image M11. The control unit 220 calculates shape information (depth information) related to the shape of the surface of the target S based on information related to the sensor 150 and the flat area SAB of the captured image M11.

[0093] In particular, the information processing device 200A according to the first embodiment acquires a flat area SAB from a captured image M11 obtained from reflected light of light IA and IB simultaneously irradiated onto a target S from a plurality of point light sources 160A and 160B.

[0094] In this way, by the information processing device 200A acquiring the flat area SAB, the assumptions 1 and 2 can be satisfied in the formula conversion in the depth calculation unit 223, and the deterioration of the accuracy of the depth calculation can be suppressed. This allows the information processing device 200A to improve the accuracy of the shape measurement.

[0095] 3. Second embodiment In the first embodiment, the information processing device 200A calculates the depth based on the captured image M11 captured by the microscope 100 while the light IA and IB are irradiated from the multiple point light sources 160A and 160B at the same time. In addition to the above example, the information processing device 200A may calculate the depth based on the captured image captured by the microscope 100 while light is irradiated from one point light source. Therefore, in the second embodiment, an example will be described in which the information processing device 200A calculates the depth based on the captured image obtained from the reflected light of the light irradiated from one of the multiple point light sources 160A and 160B and obtained for each of the point light sources 160A and 160B. Note that the information processing device 200A according to the second embodiment has the same configuration and operation as the information processing device 200A according to the first embodiment, except for the operation of the acquisition unit 221 and the area acquisition unit 225, and therefore the same reference numerals are used and some explanations are omitted.

[0096] FIG. 15 is a diagram for explaining captured images M21 and M22 acquired by the acquisition section 221 according to the second embodiment of the present disclosure.

[0097] 15, the microscope 100 first irradiates light IA from a point light source 160A to capture an image of the target S. At this time, for example, if the rough shape of the surface of the target S is not flat, an occlusion (a region where light does not reach) occurs on the surface of the target S.

[0098] For example, light IA emitted from the light source 160A hits an area SA1 on the surface of the target S but does not hit an area SA2. In this case, the acquisition unit 221 acquires a captured image M21 in which the area SA2 is darker than the area SA1, for example, as shown in FIG.

[0099] Next, the microscope 100 irradiates the object S with light IB from the point light source 160B and captures the object S. At this time, an occlusion (a region not hit by light) occurs on the surface of the object S, similar to the case where the object S is irradiated with light IA from the point light source 160A.

[0100] For example, light IB emitted from the light source 160B hits an area SB1 but not an area SB2 on the surface of the target S. In this case, the acquisition unit 221 acquires a captured image M22 in which the area SB2 is darker than the area SB1, as shown in FIG.

[0101] In this manner, the acquisition unit 221 acquires captured images M21 and M22 captured by the microscope 100 by sequentially turning on the two point light sources 160A and 160B.

[0102] The region acquisition unit 225 acquires the flat region SAB from the captured images M21 and M22 acquired by the acquisition unit 221. As described above, the light IA and IB from both the point light sources 160A and 160B hits the flat region SAB. In other words, an occlusion region that is not hit by at least one of the light IA and IB becomes a non-flat region. In this way, in the captured images M21 and M22, the light IA and IB from the point light sources 160A and 160B hit the flat region SAB, so that the luminance values ​​of the pixels in the flat region SAB in both captured images M21 and M22 are approximately the same.

[0103] Therefore, the region acquiring unit 225 acquires the flat region SAB based on the luminance values ​​of the captured images M21 and M22. The region acquiring unit 225 extracts a region in which there is no change in luminance value or the change is small (within a threshold value th2) in the corresponding pixels of the captured images M21 and M22 as the flat region SAB.

[0104] A specific method of acquiring the flat region SAB by the region acquiring section 225 will be described with reference to Fig. 16. Fig. 16 is a diagram for explaining acquisition of the flat region by the region acquiring section 225 according to the second embodiment of the present disclosure.

[0105] The region acquisition unit 225 calculates, for example, a difference D(x, y)=L1(x, y)-L2(x, y) between the luminance values ​​of corresponding pixels (x, y) of the captured images M21 and M22. Note that L1(x, y) is the luminance value of the pixel (x, y) of the captured image M21, and L2(x, y) is the luminance value of the pixel (x, y) of the captured image M22. FIG. 16 is a graph showing the relationship between the difference D and x when the y of a pixel is fixed to a predetermined value. Note that, as shown in FIG. 15, the horizontal direction of the captured images M21 and M22 is the x direction, and the vertical direction is the y direction.

[0106] For example, in area SB2 shown in FIG. 16, the luminance value of captured image M21 is high and the luminance value of captured image M22 is low (see FIG. 15). Therefore, the value of difference D in area SB2 is a large positive value. On the other hand, in area SA2 shown in FIG. 16, the luminance value of captured image M21 is low and the luminance value of captured image M22 is high (see FIG. 15). Therefore, the value of difference D in area SA2 is a large negative value. Moreover, in areas other than areas SB2 and SA2, the luminance values ​​of both captured images M21 and M22 are similarly high, so the value of difference D is close to zero.

[0107] Therefore, the region acquiring unit 225 determines the region SB2 where the difference D is equal to or greater than the threshold th2 as an exclusion region where depth calculation is not performed. The region acquiring unit 225 also determines the region SA2 where the difference D is equal to or less than the threshold -th2 as an exclusion region where depth calculation is not performed. On the other hand, the region acquiring unit 225 determines the region where the difference D is within the range of the threshold ±th2 as a processing region where depth calculation is performed. The region acquiring unit 225 acquires the region determined to be the processing region as an extraction region (flat region SAB) to be extracted from the captured images M21 and M22.

[0108] More specifically, the region acquiring unit 225 performs a threshold determination on the absolute value abs(D) of the difference D described above, and generates a mask image M12 shown in FIG. 8, for example, by making a region including pixels whose absolute value abs(D) of the difference D is equal to or greater than a threshold th2 a black region and a region including pixels whose absolute value abs(D) is less than the threshold th2 a white region. The region acquiring unit 225, for example, compares the mask image M12 with the captured image M21, and extracts pixels in which the mask image M12 is white at the same coordinates from the captured image M21, thereby acquiring a flat region SAB. Alternatively, the region acquiring unit 225 may acquire the flat region SAB by comparing the mask image M12 with the captured image M22. Note that the processing after generating the mask image M12 is the same as that in the first embodiment.

[0109] In the second embodiment, imaging is performed by irradiating one of the point light sources 160A and 160B. In this case, the amount of light incident on the target S can be kept low compared to the case of simultaneous irradiation as in the first embodiment described above, so that the overall brightness of the captured images M21 and M22 can be kept low and the contrast can be kept high. Therefore, the difference between the luminance value of the area where the light hits the captured images M21 and M22 and the luminance value of the occlusion area where the light does not hit can be increased. This allows the area acquisition unit 225 to more easily acquire the flat area SAB from the captured images M21 and M22.

[0110] On the other hand, the information processing device 200A according to the first embodiment only needs to acquire one captured image M11, so that the image acquisition time can be reduced to half the time of the second embodiment. In addition, if the target S or the sensor 150 moves while the captured images M21 and M22 are captured in sequence, the area acquisition unit 225 needs to perform processing such as alignment of the captured images M21 and M22 before acquiring the flat area SAB. Therefore, when performing depth calculation in real time, it is desirable to perform depth calculation using the information processing device 200A according to the first embodiment, which has a short image acquisition time and does not require processing such as alignment.

[0111] As described above, the information processing device 200A according to the second embodiment acquires, for each of the point light sources 160A and 160B, captured images M21 and M22 obtained from reflected light of light IA and IB irradiated onto the target S from one of the multiple point light sources 160A and 160B. The information processing device 200A extracts, as a flat region SAB, a region in which the change in luminance value (absolute value of difference D) between the multiple captured images M21 and M22 acquired for each of the point light sources 160A and 160B is less than a predetermined threshold th2.

[0112] This allows the information processing device 200A to increase the contrast of the captured images M21 and M22, improve the extraction accuracy of the flat area SAB, and improve the accuracy of shape measurement.

[0113] Although the microscope 100 is described here as having two point light sources 160A and 160B, the number of point light sources is not limited to two. For example, the microscope 100 may have three or more point light sources.

[0114] In this case, the microscope 100 sequentially lights up three or more point light sources to capture an image of the target S. The information processing device 200A acquires a plurality of captured images captured by the microscope 100. The information processing device 200A acquires, as a flat region SAB, a region in which there is little change in luminance value between the acquired plurality of captured images, in other words, a region in which the absolute value of the difference in luminance value between corresponding pixels in the plurality of captured images is less than a predetermined threshold value th2.

[0115] Even if the microscope 100 has three or more point light sources, it is not necessary to sequentially turn on all the point light sources to capture an image of the target S. For example, if the microscope 100 has three point light sources, two of the three point light sources may be sequentially turned on to capture an image. In this case, the microscope 100 may select and turn on, for example, the most distant point light source from among the multiple point light sources.

[0116] <4. Third embodiment> In the above first and second embodiments, the case where the flat region SAB is acquired from the captured image acquired by the acquisition unit 221 has been described. In addition to the above examples, the information processing device 200A may acquire the flat region SAB after performing a smoothing process on the captured image. Therefore, in the third embodiment, an example in which the information processing device 200A performs a smoothing process before acquiring the flat region SAB from the captured image will be described. Note that the information processing device 200A according to the third embodiment has the same configuration and operation as the information processing device 200A according to the first embodiment, except for the operation of the region acquisition unit 225, and therefore the same reference numerals are used and some explanations will be omitted.

[0117] FIG. 17 is a diagram for explaining the smoothing process by the area acquisition section 225 according to the third embodiment of the present disclosure.

[0118] 17, the captured image M31 acquired by the acquisition unit 221 includes local unevenness and noise of the target S. Note that the local unevenness of the target S is a calculation target of depth information in the depth calculation unit 223.

[0119] If the area acquisition section 225 attempts to perform threshold determination on a pixel-by-pixel basis on the captured image M31 shown in FIG. 17, it will be affected by the above-mentioned unevenness and noise, and the acquisition accuracy of the flat area SAB will decrease.

[0120] Therefore, before performing threshold determination to obtain the flat region SAB, the region obtaining unit 225 performs a smoothing process on the captured image M31 to generate a smoothed image M32 as shown in the lower diagram of Fig. 17. Specifically, the region obtaining unit 225 generates the smoothed image M32 by applying a low-pass filter to the captured image M31. The region obtaining unit 225 performs threshold determination on the smoothed image M32 to generate a mask image M12 shown in Fig. 8.

[0121] The region acquiring section 225, for example, compares the mask image M12 with the captured image M31 and extracts pixels at the same coordinates where the mask image M12 is white from the captured image M31, thereby acquiring the flat region SAB in the captured image M31.

[0122] In this way, the region acquiring unit 225 generates the mask image M12 based on the smoothed image M32, thereby making it possible to generate the mask image M12 while reducing the influence of fine irregularities and noise in the captured image M31. Furthermore, the region acquiring unit 225 uses the mask image M12 to acquire the flat region SAB of the captured image M31, making it possible to extract the flat region SAB including localized irregularities, and to accurately calculate the depth of the target S.

[0123] As described in the second embodiment, when the acquisition unit 221 acquires a plurality of captured images, the region acquisition unit 225 performs a smoothing process on all the acquired captured images. This allows the information processing device 200A to suppress a decrease in the acquisition accuracy of the flat region SAB and improve the accuracy of shape measurement.

[0124] <5. Fourth embodiment> In the above first and second embodiments, the area acquisition unit 225 acquires the flat area SAB by threshold-determining the luminance value of a pixel. In addition to the above examples, the information processing device 200A may acquire the flat area by dividing the captured image into a plurality of blocks. In the fourth embodiment, an example in which the information processing device 200A divides the captured image into a plurality of blocks will be described. Note that the information processing device 200A according to the fourth embodiment has the same configuration and operation as the information processing device 200A according to the first embodiment, except for the operation of the area acquisition unit 225, and therefore the same reference numerals are used and some explanations will be omitted.

[0125] FIG. 18 is a diagram for explaining acquisition of a flat region by the region acquisition section 225 according to the fourth embodiment of the present disclosure.

[0126] 18, the region acquiring unit 225 divides the captured image M41 acquired by the acquiring unit 221 into, for example, a plurality of blocks, and acquires one of the divided blocks M42 as a flat region. The region acquiring unit 225 determines the size of the block based on information about the microscope 100, and divides the captured image M41 into blocks of the determined size.

[0127] As shown in Fig. 19, even if the change in the shape of the target S is large in the entire captured image M41, the local region SC can be regarded as being generally flat. Note that Fig. 19 is a diagram for explaining acquisition of a flat region by the region acquisition unit 225 according to the fourth embodiment of the present disclosure.

[0128] For example, if the target S is human skin, empirically, if the area is 5 mm×5 mm or less, it may be considered to be a plane that satisfies the above-mentioned assumptions 1 and 2. Therefore, the area acquisition unit 225 divides the captured image M41, for example, into blocks of 5 mm×5 mm or less. The actual length P [μm] of one pixel of the captured image is calculated based on the focal length f [mm] of the sensor 150, the length d [mm] of the housing (head mount unit 10), and the pixel pitch p [μm] of the sensor 150, as described above.

[0129] For example, as described above, when the captured image M41 is divided into blocks of 5 mm×5 mm or less, the width w [pixels] of the blocks is w=5000 / P=5000f / (d×p). Specifically, when the focal length f of the sensor 150 is f=16 [mm], the length d of the head mount unit 10 is d=100 [mm], and the pixel pitch p of the sensor 150 is p=5 [μm], the length P per pixel is P=31.25 [μm]. Therefore, when the captured image M41 is divided into blocks of 5 mm×5 mm or less, the block size is 160×160 [pixels].

[0130] The normal calculation unit 222 and the depth calculation unit 223 calculate normal information and depth information for all the divided blocks. The depth calculation process is the same as the calculation process shown in FIG. 13, except for the number of blocks (flat regions) to be calculated.

[0131] In this way, in the fourth embodiment, the information processing device 200A divides the captured image M41 into a plurality of blocks to calculate the depth. This allows the information processing device 200A to acquire a flat area that satisfies assumptions 1 and 2 in the above-mentioned formulas (1) to (3), and allows accurate depth calculation. In this way, the information processing device 200A according to the fourth embodiment can improve the accuracy of shape measurement.

[0132] Here, the size of the block is 5 mm x 5 mm, but is not limited to this. For example, the imaging target of the microscope 100 may be something other than human skin. Since the size of the block that can be considered flat varies depending on the imaging target, the size of the block may be different depending on the imaging target.

[0133] For example, when the microscope 100 captures images of multiple types of objects, a table in which the types of objects to be captured correspond to appropriate block sizes is stored in the storage unit 230. In this case, the information processing device 200A selects the block size according to the type of object to be captured. Alternatively, the user may specify the block size. In this way, the block size is not limited to the above example, and various sizes can be selected.

[0134] <6. Fifth embodiment> In the above fourth embodiment, the area acquisition unit 225 acquires a flat area by dividing a captured image into blocks. In addition to the above example, the information processing device 200A may acquire a flat area according to the contrast value of the divided blocks. In the fifth embodiment, an example in which the information processing device 200A acquires a flat area according to the contrast value of the divided blocks will be described. Note that the information processing device 200A according to the fifth embodiment has the same configuration and operation as the information processing device 200A according to the fourth embodiment, except for the operation of the area acquisition unit 225, and therefore the same reference numerals are used and some explanations are omitted.

[0135] FIG. 20 is a diagram for explaining the depth of field of the sensor 150. The sensor 150 (camera) has a depth of field (a depth range in which the subject can be photographed without blurring) according to the aperture of the lens. The optical system of the camera has a focal length f at which the subject can be photographed with the highest resolution and clarity by focusing. For example, the surface M52 shown in FIG. 20 is a focused surface, that is, a surface that is in focus and can photograph the subject with the highest resolution and clarity. The range from the surface M53 to the surface M51 is the depth range (depth of field) in which the subject can be photographed without blurring.

[0136] When the head mount unit 10 is brought into contact with the target S to capture an image as in the microscope 100, the distance to the target S is shortened, resulting in a reduced amount of light incident on the sensor 150. Therefore, in order to increase the amount of light incident on the sensor 150 as much as possible, the microscope 100 captures an image with a small aperture value F, i.e., with the aperture fully open. Therefore, as shown in FIG. 21, the depth of field P1 becomes shallow. Note that FIG. 21 is a diagram for explaining the acquisition of a flat region by the region acquisition unit 225 according to the fifth embodiment of the present disclosure.

[0137] 21, since the depth of field P1 of the microscope 100 is shallow, the object S included in the depth of field P1, i.e., the surface SD1 of the object S that is roughly the same distance from the sensor 150, can be captured without blurring. On the other hand, the object S that is not included in the depth of field P1, i.e., the surfaces SD2 and SD3 that are different in distance from the sensor 150 from the surface SD1, are captured in a blurred image.

[0138] Therefore, the region acquisition unit 225 acquires an imaging region included in the depth of field P1 as a flat region. The region acquisition unit 225 acquires a flat region by acquiring an in-focus and unblurred region from the captured image. This point will be described with reference to FIG. 22. FIG. 22 is a diagram for explaining acquisition of a flat region by the region acquisition unit 225 according to the fifth embodiment of the present disclosure.

[0139] The region acquiring section 225 divides the captured image M54 acquired by the acquiring section 221 into a plurality of blocks. Note that the size of the blocks to be divided here may be the same as or different from the size of the blocks in the above-described fourth embodiment.

[0140] Next, the region acquisition unit 225 calculates a contrast value for each divided block. The contrast value "contrast" is a value obtained by contrast=(Lmax-Lmin) / (Lmax+Lmin). Note that Lmax is the maximum value of the luminance value L in the block, and Lmax=max(L(x, y)). Also, Lmin is the minimum value of the luminance value L in the block, and Lmin=min(L(x, y)).

[0141] A region that is in focus and not blurred has a high contrast value, and a region that is out of focus and not blurred has a low contrast value. Therefore, the region acquisition unit 225 acquires a flat region based on the calculated contrast value. Specifically, the region acquisition unit 225 compares the calculated contrast value with a threshold value th3 for each block. As shown in FIG. 22, the region acquisition unit 225 extracts a block whose contrast value is equal to or greater than the threshold value th3 as a flat region for performing depth calculation.

[0142] The normal calculation unit 222 and the depth calculation unit 223 calculate normal information and depth information of the flat region acquired by the region acquisition unit 225. The calculation method is the same as that in the first embodiment. The normal calculation unit 222 and the depth calculation unit 223 may calculate the normal information and depth information for each block. Alternatively, the normal calculation unit 222 and the depth calculation unit 223 may calculate the normal information and depth information for a region extracted by the region acquisition unit 225 as a flat region, that is, a plurality of blocks having a contrast value contrast equal to or greater than a threshold value th3, all together.

[0143] As described above, in the fifth embodiment, the information processing device 200A divides the captured image M54 into a plurality of blocks. The information processing device 200A obtains flat regions according to the contrast value "contrast" calculated for each divided block.

[0144] In this way, the information processing device 200A can calculate the depth in the flat region that satisfies assumptions 1 and 2 in the above-mentioned formulas (1) to (3) by acquiring the flat region according to the contrast value contrast of the captured image M54. This allows the information processing device 200A according to the fifth embodiment to improve the accuracy of shape measurement.

[0145] <7. Sixth embodiment> In the fifth embodiment, the area acquisition unit 225 divides one captured image into a plurality of blocks and acquires a flat area according to the contrast value of each divided block. In addition to the above example, the information processing device 200A may divide a plurality of captured images having different subject depths into a plurality of blocks and acquire a flat area according to the contrast value of each divided block. Therefore, in the sixth embodiment, an example in which the information processing device 200A acquires a flat area according to the contrast value of each block of a plurality of captured images will be described. Note that the information processing device 200A according to the sixth embodiment has the same configuration and operation as the information processing device 200A according to the fifth embodiment, except for the operation of the acquisition unit 221 and the area acquisition unit 225, and therefore the same reference numerals are used and some explanations are omitted.

[0146] FIG. 23 is a diagram for explaining a plurality of captured images acquired by the acquisition unit 221 according to the sixth embodiment of the present disclosure. The acquisition unit 221 acquires a plurality of captured images having different depths of field. As shown in FIG. 23, the microscope 100 acquires a plurality of captured images having different depths of field (focus planes) by moving the sensor 150 of the microscope 100. In the example of FIG. 23, the microscope 100 captures three captured images by capturing an image of the target S three times while moving the sensor 150 up and down. For example, the microscope 100 captures an image having a depth of field closest to the sensor 150 in the first time, and captures an image having a depth of field farthest from the sensor 150 in the third time. For example, the microscope 100 captures an image having a depth of field intermediate between the first and second times in the second time.

[0147] Note that the depth of field and the number of captured images shown in Fig. 23 are merely examples and are not limited thereto. For example, the microscope 100 may capture two captured images or four or more captured images. In addition, the range of the depth of field during each capture may be continuous or may partially overlap.

[0148] The acquisition unit 221 controls, for example, the microscope 100 to acquire captured images M61 to M63 each having a different subject depth.

[0149] Fig. 24 is a diagram for explaining acquisition of a flat region by the region acquisition unit 225 according to the sixth embodiment of the present disclosure. Here, the acquisition unit 221 controls the microscope 100 to capture captured images having different subject depths as shown in Fig. 23, thereby acquiring captured images M61 to M63 as shown in Fig. 24.

[0150] The region acquiring section 225 divides each of the captured images M61 to M63 into blocks. The region acquiring section 225 calculates a contrast value "contrast" for each divided block. The region acquiring section 225 compares the calculated contrast value "contrast" of each block with a threshold value th3, and extracts blocks whose contrast value "contrast" is equal to or greater than the threshold value th3 as flat regions.

[0151] 24, the region acquiring section 225 extracts a region M61A including blocks in the captured image M61 where the contrast value "contrast" is equal to or greater than the threshold value th3 from the captured image M61. Similarly, the region acquiring section 225 extracts a region M62A including blocks in the captured image M62 where the contrast value "contrast" is equal to or greater than the threshold value th3 from the captured image M62. Also, the region acquiring section 225 extracts a region M63A including blocks in the captured image M63 where the contrast value "contrast" is equal to or greater than the threshold value th3 from the captured image M63.

[0152] Finally, the region acquiring section 225 synthesizes the extracted regions M61A to M63A to generate an image M64 including a flat region. In other words, the region acquiring section 225 synthesizes the in-focus and non-blurred regions of the captured images M61 to M63 to generate the image M64.

[0153] Note that the image M64 is regarded as a flat region, and the processes executed by the normal calculation unit 222 and the depth calculation unit 223 are the same as those in the fifth embodiment.

[0154] As described above, the information processing device 200A according to the sixth embodiment acquires a plurality of captured images having different depths of field. The information processing device 200A divides each of the acquired captured images into a plurality of blocks. The information processing device 200A acquires a flat region according to the contrast value (contrast) of each divided block.

[0155] As a result, the information processing device 200A can calculate the depth in the flat region that satisfies assumptions 1 and 2 in the above-mentioned formulas (1) to (3) by acquiring the flat region according to the contrast value contrast. Also, the information processing device 200A can expand the flat region according to the contrast value contrast by acquiring the flat region from a plurality of captured images.

[0156] Note that, although the region acquisition unit 225 has been described here as combining the regions M61A to M63A, this is not limiting. For example, the normal calculation unit 222 and the depth calculation unit 223 may calculate normal information and depth information for each of the regions M61A to M63A, respectively, and when the display control unit 224 causes the display unit to display the results, the combined result of the regions M61A to M63A may be displayed.

[0157] <8. Seventh embodiment> In the above first to sixth embodiments, the area acquisition unit 225 acquires a flat area from a captured image. In addition to the above examples, high-frequency components may be extracted from normal information calculated by the normal calculation unit 222. In the seventh embodiment, an example in which high-frequency components are extracted from normal information to calculate depth information will be described.

[0158] Fig. 25 is a diagram showing a configuration example of an information processing device 200B according to a seventh embodiment of the present disclosure. The information processing device 200B shown in Fig. 25 has the same components as the information processing device 200A shown in Fig. 6, except that a control unit 220B does not include a region acquisition unit 225 and includes a normal frequency separation unit 226B.

[0159] The normal calculation unit 222 calculates normal information for each pixel of the captured image acquired by the acquisition unit 221 .

[0160] The normal frequency separation unit 226B separates high-frequency components (hereinafter also referred to as normal high frequencies) from the normal information. Here, as described above, the depth information to be calculated by the depth calculation unit 223 is information on the local uneven shape of the target S. In other words, the depth information calculated by the depth calculation unit 223 is high-frequency components of the shape of the surface of the target S, and does not include low-frequency components. The normal frequency separation unit 226 removes low-frequency components, which are normal information of a rough shape, from the normal information to generate normal high frequencies. Since the normal high frequencies do not include low-frequency components (normal information of a rough shape), they satisfy assumptions 1 and 2 of the above-mentioned formulas (1) to (3).

[0161] FIG. 26 is a diagram for explaining frequency separation by the normal frequency separation section 226B according to the seventh embodiment of the present disclosure.

[0162] For example, assume that normal information X as shown in the upper diagram of Fig. 26 is input to the normal frequency separation unit 226B. Note that in Fig. 26, the actual surface of the object S is indicated by a dotted line, the general shape of the surface of the object S is indicated by a solid line, and the normal to the surface of the object S is indicated by an arrow.

[0163] In this case, the normal frequency separation unit 226B separates the normal information into high-frequency and low-frequency components using, for example, a convolution filter F(X) and extracts the normal high-frequency components Y shown in the center diagram of Fig. 26. The convolution filter F(X) is a filter designed in advance to extract high-frequency components when normal information is input.

[0164] The depth calculation unit 223 can calculate depth information Z (see the lower diagram in Figure 26) that satisfies assumptions 1 and 2 by converting the normal high frequencies separated by the normal frequency separation unit 226B using the above-mentioned equations (1) to (3), etc.

[0165] In this way, the information processing device 200B extracts high-frequency components from the normal information, and calculates depth information based on the extracted high-frequency components.

[0166] This allows the information processing device 200B to perform equation conversion using the above-mentioned equations (1) to (3) etc. while satisfying assumptions 1 and 2, thereby suppressing a decrease in the calculation accuracy of depth information. Therefore, the information processing device 200B can improve the accuracy of shape measurement compared to a case where frequency separation is not performed.

[0167] <9. Eighth embodiment> In the seventh embodiment, the depth information is calculated from the high frequency components of the normal information. In addition to the above example, the depth information may be calculated using low frequency components in addition to the high frequency components of the normal information. Therefore, in the eighth embodiment, an example in which the high frequency components and low frequency components are separated from the normal information to calculate the depth information will be described.

[0168] Fig. 27 is a diagram showing a configuration example of an information processing device 200C according to an eighth embodiment of the present disclosure. The information processing device 200C shown in Fig. 27 has the same components as the information processing device 200B shown in Fig. 25, except for a normal frequency separation unit 226C and a depth calculation unit 223C of a control unit 220C.

[0169] The normal frequency separation unit 226C separates high-frequency components and low-frequency components (hereinafter also referred to as normal low-frequency components) from the normal information and outputs them to the depth calculation unit 223C. As described above, the local uneven shape of the object S contributes to the high-frequency components of the normal information. In addition, the general shape of the object S contributes to the low-frequency components of the normal information. This point will be described with reference to FIG. 28.

[0170] FIG. 28 is a diagram for explaining frequency separation by the normal frequency separation section 226C according to the eighth embodiment of the present disclosure.

[0171] For example, assume that normal information X as shown in Fig. 28(a) is input to the normal frequency separation unit 226C. In Fig. 28, the actual surface of the object S is indicated by a dotted line, the general shape of the surface of the object S is indicated by a solid line, and the normal to the surface of the object S is indicated by an arrow.

[0172] In this case, the normal frequency separation unit 226C extracts high frequency components of normal information using, for example, a convolution filter F(X) to extract high frequency normal YH=F(X). As shown in the upper diagram of FIG. 28(b), the surface shape represented by the high frequency normal YH has a rough shape (low frequency normal YL) removed, resulting in a shape with localized unevenness on a flat surface. The convolution filter F(X) is a filter designed in advance to extract high frequency components when normal information is input.

[0173] In addition, the normal frequency separation unit 226C extracts the normal low frequency YL by subtracting the normal high frequency YH from the normal information X. That is, the normal frequency separation unit 226C extracts the normal low frequency YL by calculating the normal low frequency YL=X-YH=XF(X). As shown in the lower diagram of FIG. 28(b), the surface shape represented by the normal low frequency YL is a rough shape with local unevenness removed.

[0174] The depth calculation unit 223C performs equation conversion on each of the normal high frequency YH and the normal low frequency YL, and calculates the high frequency component ZH (hereinafter also referred to as depth high frequency) and low frequency component ZL (hereinafter also referred to as depth low frequency) of the depth information, as shown in Figure 28(c).

[0175] Here, since the normal high frequency YH satisfies assumptions 1 and 2 of the equation conversion as described above, the depth calculation unit 223C can calculate the deep high frequency ZH with high accuracy.

[0176] The depth calculation unit 223C calculates the depth low frequency ZL by performing equation conversion of the normal low frequency YL with λ and μ in the above-mentioned equations (1) to (3) set to zero (λ=0, μ=0). In this way, by setting the coefficient of the cost term to zero, the depth low frequency ZL can be calculated in a state in which assumptions 1 and 2 are eliminated. The normal low frequency YL is a low-frequency component of the normal information X from which the normal high frequency YH has been removed, and is normal information that represents a state in which local uneven shapes have been removed from the surface shape of the target S. Therefore, even if the depth calculation unit 223C performs equation conversion of the normal low frequency YL in a state in which assumptions 1 and 2 are eliminated, the depth low frequency ZL can be calculated with high accuracy.

[0177] The depth calculation unit 223C calculates the depth information Z shown in FIG. 28(d) by combining the calculated depth high frequency signal ZH and depth low frequency signal ZL.

[0178] In this way, the information processing device 200C separates the normal information calculated from the captured image into frequencies. The information processing device 200C calculates depth information for each of the separated frequencies, and calculates the depth information of the object S by combining the calculated depth information.

[0179] This allows the information processing device 200C to calculate the depth of the rough shape in addition to the depth of the local uneven shape. Moreover, the information processing device 200C can calculate the depth information of the object S with higher accuracy by combining these depths.

[0180] Here, the depth calculation unit 223C converts the normal low frequency YL by setting λ and μ in the above-mentioned formulas (1) to (3) to zero (λ=0, μ=0), but is not limited to this. It is sufficient that the depth calculation unit 223C can calculate the depth low frequency ZL in a state where the influence of assumptions 1 and 2 is small, and the values ​​of λ and μ may be set to values ​​small enough that assumptions 1 and 2 do not affect the calculation of the depth low frequency ZL.

[0181] <10. Ninth embodiment> In the first embodiment, the learning device 300 is trained using the image M16, which is a true value. In addition to the above example, the information processing device 200A may train the learning device 300 using an image obtained by shifting the image M16 as the correct answer data. Therefore, in the ninth embodiment, an example in which the information processing device 200A trains the learning device 300 using an image obtained by shifting the image M16 as the correct answer data will be described with reference to FIG. 29. FIG. 29 is a diagram for describing a learning method of the learning device 300 according to the ninth embodiment. Note that the information processing device 200A according to the ninth embodiment has the same configuration and operation as the information processing device 200A according to the first embodiment, except for the operation of the normal calculation unit 222, and therefore the same reference numerals are used and some explanations are omitted.

[0182] As described above, the normal calculation unit 222 performs learning by updating the weights of the learning device 300 using the captured image M14 as input data and the normal image M16 as answer data (true value). Here, the captured image M14 is captured by a device that captures RGB images, such as the microscope 100. On the other hand, the normal image M16 is generated based on shape information measured by a device that directly measures the surface shape of the target S, such as a non-contact 3D measuring instrument.

[0183] In this way, if the optical system of the device that acquires the input image is different from the optical system of the device that acquires the correct image, it becomes difficult to align the input image with the correct image. Therefore, the output image and the correct image obtained by inputting the input image to the learning device 300 may be shifted by several pixels. In this way, if the learning device 300 is trained in a state where the output image and the correct image are shifted, a learning device 300 that outputs an output image that is blurred with respect to the input image is generated.

[0184] 29, the normal calculation unit 222 shifts the normal image M16 in the x and y directions during learning to determine the filter coefficients (weights) of the learning device (CNN) 300, and calculates the similarity between the shifted normal image M16 and the output image M15. Note that the normal calculation unit 222 shifts the normal image M16 in the x direction within the range of 0 to ±x, and in the y direction within the range of 0 to ±y.

[0185] For example, the normal calculation unit 222 calculates the least square error between the normal image M16 after the shift and the output image M15 for each shift amount. The normal calculation unit 222 sets the minimum value of the calculated least square errors as the final loss and updates the weight of the learning device 300. Note that instead of the minimum square error, the minimum average error between the normal image M16 after the shift and the output image M15 may be calculated for each shift amount, and the minimum value of the calculated minimum average errors may be set as the final loss.

[0186] In this way, the learning device 300 updates the weights according to the similarity between a shifted image (shifted normal image) obtained by shifting the true image (normal image) and the output image M15. More specifically, the learning device 300 updates the weights according to the similarity between each of a plurality of shifted images (shifted normal images, also called correct answer candidate images) obtained by shifting the true image (normal image) by different shift amounts (number of pixels) and the output image M15.

[0187] As a result, even if the output image M15 of the learning device 300 and the normal image M16 are not aligned with each other, the weights of the learning device 300 can be updated with the positions of both images in place. This can further improve the learning accuracy of the learning device 300. In addition, by the normal calculation unit 222 calculating normal information using the learning device 300, the normal information can be calculated with higher accuracy, and the accuracy of shape measurement of the information processing device 200A can be further improved.

[0188] <11. Tenth embodiment> In the above-described first embodiment, a case where a flat region is acquired from a captured image has been described. In addition to the above-described example, the information processing device 200A may divide a captured image into a plurality of regions and acquire a flat region for each divided region. In the tenth embodiment, therefore, an example in which the information processing device 200A divides a captured image into a plurality of regions and acquires a flat region for each divided region will be described with reference to FIG. 30. FIG. 30 is a diagram showing a configuration example of a control unit 220D of the information processing device 200A according to the tenth embodiment of the present disclosure.

[0189] 30, a control unit 220D of an information processing device 200A is the same as the control unit 220 shown in FIG. 6, except that a region acquisition unit 225D is provided instead of the region acquisition unit 225, and a depth stacking unit 227 is further provided.

[0190] 30, the region acquiring section 225D divides the captured image M71 acquired by the acquiring section 221 into three divided regions M71A to M71C. Note that the number of regions divided by the region acquiring section 225D is not limited to three, and may be two, four or more.

[0191] The region acquisition unit 225D acquires divided flat regions M72A-M72C for each of the divided regions M71A-M71C. The normal calculation unit 222 calculates normal region images M73A-M73C based on the divided flat regions M72A-M72C, respectively, and the depth calculation unit 223 calculates depth region images M74A-M74C based on the normal region images M73A-M73C.

[0192] The depth synthesis unit 227 synthesizes the depth region images M74A to M74C to generate a depth image M75, and outputs the depth image M75 to the display control unit 224.

[0193] The depth stacking unit 227 may also stack the divided flat regions M72A to M72C and the normal region images M73A to M73C in addition to the depth region images M74A to M74. The processing in each unit may be performed sequentially for each region, or may be performed in parallel for each region.

[0194] In this way, the information processing device 200A divides the captured image into a plurality of divided regions M71A to M71C, and obtains divided flat regions M72A to M72C for each of the divided regions M71A to M71C. In addition, the information processing device 200A calculates normal information and depth information for each of the divided flat regions M72A to M72C.

[0195] This makes it possible to reduce the size of the captured image (area) to be processed by each unit, thereby reducing the processing load and improving the processing speed.

[0196] <12. Other embodiments> The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0197] In each of the above embodiments, an example has been shown in which the head mount unit 10 is a cylindrical part mounted on the tip of the microscope 100. However, the head mount unit 10 does not necessarily have to be cylindrical in shape as long as it is a structure for keeping the distance between the target S and the sensor 150 of the microscope 100 constant.

[0198] <13. Application Examples> An application example of the information processing devices 200A to 200C according to the first to tenth embodiments will be described with reference to Fig. 31. Fig. 31 is a block diagram showing an example of a schematic configuration of a patient's in-vivo information acquisition system using a capsule endoscope to which the technology according to the present disclosure (the present technology) can be applied.

[0199] The in-vivo information acquisition system 10001 includes a capsule endoscope 10100 and an external control device 10200 .

[0200] The capsule endoscope 10100 is swallowed by a patient during an examination. The capsule endoscope 10100 has an imaging function and a wireless communication function, and while moving inside an organ such as the stomach or intestines by peristalsis or the like until it is naturally discharged from the patient, sequentially captures images of the inside of the organ (hereinafter also referred to as in-vivo images) at predetermined intervals, and sequentially wirelessly transmits information about the in-vivo images to an external control device 10200 outside the body.

[0201] The external control device 10200 comprehensively controls the operation of the in-vivo information acquisition system 10001. In addition, the external control device 10200 receives information about the in-vivo image transmitted from the capsule endoscope 10100, and generates image data for displaying the in-vivo image on a display device (not shown) based on the received information about the in-vivo image.

[0202] In this manner, the in-vivo information acquiring system 10001 can obtain in-vivo images capturing images of the state inside the patient's body at any time from when the capsule endoscope 10100 is swallowed to when it is expelled.

[0203] The configurations and functions of the capsule endoscope 10100 and the external control device 10200 will be described in more detail.

[0204] The capsule endoscope 10100 has a capsule-shaped housing 10101, which houses a light source unit 10111, an imaging unit 10112, an image processing unit 10113, a wireless communication unit 10114, a power supply unit 10115, a power supply unit 10116, and a control unit 10117.

[0205] The light source unit 10111 is composed of a light source such as an LED (Light Emitting Diode), and irradiates the imaging field of the imaging unit 10112 with light.

[0206] The imaging unit 10112 is composed of an imaging element and an optical system consisting of multiple lenses provided in front of the imaging element. Reflected light (hereinafter referred to as observation light) of light irradiated onto the body tissue to be observed is collected by the optical system and enters the imaging element. In the imaging unit 10112, the imaging element photoelectrically converts the observation light incident thereon, and generates an image signal corresponding to the observation light. The image signal generated by the imaging unit 10112 is provided to an image processing unit 10113.

[0207] The image processing unit 10113 is configured with a processor such as a CPU or a GPU (Graphics Processing Unit), and performs various signal processing on the image signal generated by the imaging unit 10112. The image processing unit 10113 provides the image signal after the signal processing to the wireless communication unit 10114 as RAW data.

[0208] The wireless communication unit 10114 performs predetermined processing such as modulation processing on the image signal that has been subjected to signal processing by the image processing unit 10113, and transmits the image signal to the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 also receives a control signal related to drive control of the capsule endoscope 10100 from the external control device 10200 via the antenna 10114A. The wireless communication unit 10114 provides the control signal received from the external control device 10200 to the control unit 10117.

[0209] The power supply unit 10115 is composed of an antenna coil for receiving power, a power regeneration circuit for regenerating power from the current generated in the antenna coil, a boost circuit, etc. In the power supply unit 10115, power is generated using the principle of so-called non-contact charging.

[0210] The power supply unit 10116 is composed of a secondary battery, and stores the power generated by the power supply unit 10115. In order to avoid cluttering the drawing, arrows and other symbols indicating the destination of the power supply from the power supply unit 10116 are omitted in Fig. 31, but the power stored in the power supply unit 10116 is supplied to the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the control unit 10117 and can be used to drive these units.

[0211] The control unit 10117 is composed of a processor such as a CPU, and appropriately controls the driving of the light source unit 10111, the imaging unit 10112, the image processing unit 10113, the wireless communication unit 10114, and the power supply unit 10115 in accordance with control signals transmitted from the external control device 10200.

[0212] The external control device 10200 is composed of a processor such as a CPU or a GPU, or a microcomputer or a control board in which a processor and a storage element such as a memory are mixed. The external control device 10200 controls the operation of the capsule endoscope 10100 by transmitting a control signal to the control unit 10117 of the capsule endoscope 10100 via the antenna 10200A. In the capsule endoscope 10100, for example, the light irradiation conditions for the observation target in the light source unit 10111 can be changed by the control signal from the external control device 10200. Furthermore, the imaging conditions (for example, the frame rate and exposure value in the imaging unit 10112) can be changed by the control signal from the external control device 10200. Furthermore, the content of the processing in the image processing unit 10113 and the conditions for the wireless communication unit 10114 to transmit an image signal (for example, the transmission interval, the number of transmitted images, etc.) may be changed by the control signal from the external control device 10200.

[0213] The external control device 10200 also applies various image processing to the image signal transmitted from the capsule endoscope 10100 to generate image data for displaying the captured in-vivo image on a display device. As the image processing, for example, various signal processing such as development processing (demosaic processing), high image quality processing (band enhancement processing, super-resolution processing, noise reduction processing, camera shake correction processing, etc.), and enlargement processing (electronic zoom processing) can be performed alone or in combination. The external control device 10200 controls the driving of the display device to display the captured in-vivo image based on the generated image data. Alternatively, the external control device 10200 may record the generated image data in a recording device (not shown) or print out the image data on a printing device (not shown).

[0214] An example of an in-vivo information acquisition system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the external control device 10200 among the above-described configurations. By applying the technology according to the present disclosure to the external control device 10200, it is possible to measure the surface shape of the inside of the body from the in-vivo image captured by the capsule endoscope 10100.

[0215] (Application example to endoscopic surgery system) The technology according to the present disclosure may be further applied to an endoscopic surgery system. Fig. 32 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0216] 32 shows a state in which an operator (doctor) 11131 is performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0217] The endoscope 11100 is composed of a lens barrel 11101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may be configured as a so-called flexible scope having a flexible lens barrel.

[0218] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0219] An optical system and an image sensor are provided inside the camera head 11102, and the reflected light (observation light) from the observation target is collected on the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal is transmitted to a CCU (camera control unit) 11201 as RAW data.

[0220] The CCU 11201 is configured with a CPU, a GPU, etc., and performs overall control of the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0221] Under the control of the CCU 11201, the display device 11202 displays an image based on an image signal that has been subjected to image processing by the CCU 11201.

[0222] The light source device 11203 is composed of a light source such as an LED (Light Emitting Diode), and supplies the endoscope 11100 with irradiation light when photographing an operation site or the like.

[0223] The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiated light, magnification, focal length, etc.) of the endoscope 11100.

[0224] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 sends gas into the body cavity of the patient 11132 via the insufflation tube 11111 to inflate the body cavity for the purpose of securing the field of view of the endoscope 11100 and securing the working space of the surgeon. The recorder 11207 is a device capable of recording various information related to the surgery. The printer 11208 is a device capable of printing various information related to the surgery in various formats such as text, image, or graph.

[0225] The light source device 11203 that supplies irradiation light to the endoscope 11100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation object with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0226] The light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. The driving of the image sensor of the camera head 11102 may be controlled in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0227] The light source device 11203 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, it is possible to irradiate an excitation light to a body tissue and observe the fluorescence from the body tissue (autofluorescence observation), or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0228] FIG. 33 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in FIG.

[0229] The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other by a transmission cable 11400 so as to be able to communicate with each other.

[0230] The lens unit 11401 is an optical system provided at the connection portion with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

[0231] The imaging unit 11402 is composed of an imaging element. The imaging element constituting the imaging unit 11402 may be one (so-called single-plate type) or multiple (so-called multi-plate type). When the imaging unit 11402 is composed of a multi-plate type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining the image signals. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to 3D (Dimensional) display. By performing 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 11402 is composed of a multi-plate type, multiple lens units 11401 may be provided corresponding to each imaging element.

[0232] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately behind the objective lens.

[0233] The driving unit 11403 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be appropriately adjusted.

[0234] The communication unit 11404 is configured by a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 via the transmission cable 11400 as RAW data.

[0235] Furthermore, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201, and supplies the control signal to the camera head control unit 11405. The control signal includes information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value at the time of capturing the image, and / or information specifying the magnification and focus of the captured image.

[0236] The image capturing conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by a user, or may be automatically set by the control unit 11413 of the CCU 11201 based on an acquired image signal. In the latter case, the endoscope 11100 is equipped with a so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0237] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 via the communication unit 11404.

[0238] The communication unit 11411 is configured with a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

[0239] Furthermore, the communication unit 11411 transmits, to the camera head 11102, a control signal for controlling the driving of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, or the like.

[0240] The image processing unit 11412 performs various types of image processing on the image signal, which is RAW data sent from the camera head 11102 .

[0241] The control unit 11413 performs various controls related to imaging of the surgical site etc. by the endoscope 11100 and display of the captured image obtained by imaging the surgical site etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

[0242] Further, the control unit 11413 causes the display device 11202 to display the captured image showing the surgical site, etc., based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can recognize surgical tools such as forceps, specific living body parts, bleeding, mist when the energy treatment tool 11112 is used, etc., by detecting the shape and color of the edge of an object included in the captured image. When the control unit 11413 causes the display device 11202 to display the captured image, it may use the recognition result to superimpose various types of surgery support information on the image of the surgical site. By superimposing and presenting the surgery support information to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

[0243] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable for communication of electrical signals, an optical fiber for optical communication, or a composite cable of these.

[0244] Here, in the example of FIG. 33, communication is performed wired using the transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may be performed wirelessly.

[0245] An example of an endoscopic surgery system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the CCU 11201 among the configurations described above. Specifically, the above-mentioned control units 220, 220B, and 220C can be applied to the image processing unit 11412. By applying the technology according to the present disclosure to the image processing unit 11412, it is possible to measure the surface shape of the inside of the body from the in-body image captured by the endoscope 11100.

[0246] Although an endoscopic surgery system has been described here as an example, the technology disclosed herein may also be applied to other systems, such as a microsurgery system.

[0247] <14. Supplementary Information> Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0248] Of the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0249] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0250] In addition, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not inconsistent. In addition, in the above-described embodiment, a microscope is used as an example of an image processing device, but the image processing of the present disclosure can be applied to imaging devices other than a microscope.

[0251] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification in addition to or in place of the above effects.

[0252] Note that the following configurations also fall within the technical scope of the present disclosure. (1) Acquire an image of the target captured by a sensor, The captured image is an image obtained from reflected light of light irradiated onto the target from a plurality of light sources disposed at different positions, Extracting a flat region from the captured image based on a luminance value of the captured image; a control unit that calculates shape information regarding a shape of a surface of the object based on information about the sensor and the flat region of the captured image; An information processing device comprising: (2) The control unit is acquiring the captured image obtained from the reflected light of the light simultaneously irradiated from the plurality of light sources onto the target; An information processing device as described in (1). (3) The control unit is A region in which the luminance value of the captured image is equal to or greater than a predetermined threshold is defined as the flat region. An information processing device according to (1) or (2). (4) The control unit is The captured image obtained from the reflected light of the light irradiated from one of the plurality of light sources onto the object is acquired for each of the light sources; extracting, as the flat region, a region in which a change in the luminance value between the plurality of captured images acquired for each of the light sources is less than a predetermined threshold value; An information processing device as described in (1). (5) The control unit is performing a smoothing process on the captured image to generate a smoothed image; extracting the flat region from the captured image based on the smoothed image; An information processing device according to any one of (1) to (4). (6) The control unit is Dividing the captured image into a plurality of divided regions; extracting the flat region for each of the plurality of divided regions and calculating shape information; An information processing device according to any one of (1) to (5). (7) The control unit is Calculating normal information of the flat region in the captured image; inputting the normal information into a model generated by machine learning to obtain the shape information; An information processing device according to any one of (1) to (6). (8) The model is The weights are updated based on a comparison result between a plurality of correct candidate images and the output data of the model, The plurality of correct candidate images are generated by shifting the correct image by different numbers of pixels. An information processing device according to (7). (9) The model is Calculating a least square error between each of a plurality of correct candidate images and the output data; updating the weights based on a minimum value of a plurality of the least square errors; An information processing device according to (8). (10) The control unit is extracting the flat region based on a contrast value calculated from a luminance value of the captured image; An information processing device as described in (1). (11) The control unit is Dividing the captured image into a plurality of divided regions; Calculating the contrast value for each of the plurality of divided regions; extracting the flat region by determining whether the divided region is flat or not according to the contrast value; An information processing device as described in (1). (12) The control unit is acquiring a plurality of the captured images each having a different depth of field plane of the sensor; extracting the flat region from each of the plurality of captured images; An information processing device according to (10) or (11). (13) Acquire an image of the target captured by a sensor, The captured image is an image obtained from reflected light of light irradiated onto the target from a plurality of light sources disposed at different positions, Extracting a flat region from the captured image based on a luminance value of the captured image; calculating shape information relating to a shape of a surface of the object based on information about the sensor and the flat region of the captured image; Information processing methods. [Explanation of symbols]

[0253] 10 Head mount 100 Microscope 150 Sensors 160 point light source 200 Information processing device 220 Control section 221 Acquisition Department 222 Normal calculation section 223 Depth calculation section 224 Display control section 225 Area acquisition part 226 Normal Frequency Separation Unit 227 Depth synthesis section 230 Storage section

Claims

1. Acquiring an image of an object captured by a sensor, the image being obtained from reflected light of light simultaneously irradiated onto the object from a plurality of light sources disposed at different positions; extracting a flat region from the captured image, the flat region having a luminance value of the captured image equal to or greater than a predetermined threshold value, based on the luminance value of the captured image; a control unit that calculates shape information regarding a shape of a surface of the object based on information about the sensor and the flat region of the captured image; An information processing device comprising:

2. The control unit is The captured image obtained from the reflected light of the light irradiated from one of the plurality of light sources onto the object is acquired for each of the light sources; extracting, as the flat region, a region in which a change in the luminance value between the plurality of captured images acquired for each of the light sources is less than a predetermined threshold value; The information processing device according to claim 1 .

3. The control unit is performing a smoothing process on the captured image to generate a smoothed image; extracting the flat region from the captured image based on the smoothed image; The information processing device according to claim 1 .

4. The control unit is Dividing the captured image into a plurality of divided regions; extracting the flat region for each of the plurality of divided regions and calculating shape information; The information processing device according to claim 3 .

5. The control unit is Calculating normal information of the flat region in the captured image; inputting the normal information into a model generated by machine learning to obtain the shape information; The information processing device according to claim 4.

6. The model is The weights are updated based on a comparison result between a plurality of correct candidate images and the output data of the model, The plurality of correct candidate images are generated by shifting the correct image by different numbers of pixels. The information processing device according to claim 5 .

7. The model is Calculating a least square error between each of a plurality of correct candidate images and the output data; updating the weights based on a minimum value of a plurality of the least square errors; The information processing device according to claim 6.

8. The control unit is extracting the flat region based on a contrast value calculated from a luminance value of the captured image; The information processing device according to claim 1 .

9. The control unit is Dividing the captured image into a plurality of divided regions; Calculating the contrast value for each of the plurality of divided regions; extracting the flat region by determining whether the divided region is flat or not according to the contrast value; The information processing device according to claim 8.

10. The control unit is acquiring a plurality of the captured images each having a different depth of field plane of the sensor; extracting the flat region from each of the plurality of captured images; The information processing device according to claim 8.

11. Acquiring an image of an object captured by a sensor, the image being obtained from reflected light of light simultaneously irradiated onto the object from a plurality of light sources disposed at different positions; extracting a flat region from the captured image, the flat region having a luminance value of the captured image equal to or greater than a predetermined threshold value, based on the luminance value of the captured image; calculating shape information relating to a shape of a surface of the object based on information about the sensor and the flat region of the captured image; Information processing methods.

Images