Optical processing method, optical processing apparatus, optical processing system, and optical processing program

Abstract

To provide an optical processing method capable of acquiring object information even in a case where an object shape cannot be expressed by an analytical solution of deflection.SOLUTION: According to an embodiment, an optical processing method includes: inputting an image including direction information of light from an object as hue information for each pixel into an artificial neural network; and acquiring depth information of the object for the image based on a result output from the artificial neural network.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to an optical processing method, an optical processing device, an optical processing system, and an optical processing program. [Background technology]

[0002] In various industries, inspections are carried out by irradiating light on an object or medium, obtaining information about the object by detecting the change in direction (deflection) of the light relative to the incident direction, and analyzing the deflection. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] H. Ohno, “One-shot color mapping imaging system of light direction extracted from a surface BRDF,” OSA Continuum Vol. 3, No. 12, pp. 3343-3350 (2020). [Non-Patent Document 2] H. Ohno, “One-shot three-dimensional measurement method with the color mapping of light direction,” OSA Continuum, Vol. 4, No. 3, (2021). [Non-Patent Document 3] H. Ohno and K. Toya, “Reconstruction method of axisymmetric refractive index fields with background-oriented schlieren,” Appl. Optics, vol. 57, No. 30, 2018. [Non-Patent Document 4] M. Mattheakis, D. Sondak, AS Dogra, and P. Protopapas, “Hamiltonian neural networks for solving equations of motion,” Phys. Rev. E 105(6), 065305 (2022). Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an optical processing method, an optical processing device, an optical processing system, and an optical processing program that are capable of acquiring object information even when the object shape cannot be expressed by an analytical solution of deflection. [Brief description of the drawings]

[0005] [Figure 1] 1 is a schematic block diagram of an optical processing system according to an embodiment; [Diagram 2] FIG. 1 is a diagram showing an example of a configuration of an optical processing device according to an embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing an optical processing flow using the optical processing device according to the embodiment. [Figure 4] FIG. 2 is a schematic diagram showing an optical processing flow using the optical processing apparatus according to the first embodiment. [Diagram 5] FIG. 11 is a schematic cross-sectional view showing an optical device according to a second embodiment. [Figure 6] 1 is a diagram illustrating a schematic diagram of the relationship between incident light and reflected light incident on a surface of an object and a normal vector of the object surface. [Figure 7] FIG. 11 is a schematic diagram showing an optical processing flow in which an image acquired by the optical device according to the second embodiment is optically processed by the optical processing device according to the second embodiment. [Figure 8] 1 is an example of an image including hue information captured by an optical device of an aluminum surface. [Figure 9] An example of a wavelength selection section used in the optical setup to obtain the image shown in Figure 8. [Figure 10]An example of a 3D image including depth information obtained by inputting the image shown in FIG. 8 into an optical processor and solving the problem using an artificial neural network. [Figure 11] The distribution of inclination angles, α, of the surface of object S relative to the horizontal plane, based on the image shown in FIG. 8 and the three-dimensional image shown in FIG. [Figure 12] FIG. 11 is a schematic perspective view showing an example of an optical device according to a third embodiment in which an imaging optical element is omitted. [Figure 13] 13 is a schematic perspective view showing an example in which a wavelength selection section of the optical device shown in FIG. 12 is rotated 90° with respect to the optical axis. [Figure 14] 13 is a schematic diagram showing an optical device in which an imaging optical element is omitted, and an optical processing device according to a fourth embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Each embodiment will be described below with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as in reality. Even when the same part is shown, the dimensions and ratios may be shown differently depending on the drawing. In this specification and each drawing, elements similar to those described above with respect to the previous drawings are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0007] It is known that light can be treated as electromagnetic waves according to Maxwell's equations. In this specification, light can be visible light, X-rays, infrared rays, far-infrared rays, millimeter waves, or microwaves. In other words, electromagnetic waves of various wavelengths are referred to as light here. In particular, light in the wavelength range of about 430 nm to 750 nm is considered to be visible light, and hereinafter, visible light will be used as the term light.

[0008] (First embodiment) Hereinafter, an optical processing system 1 according to this embodiment will be described with reference to Figures 1 to 4. This embodiment is a modified example of the optical processing system 1 of the first embodiment.

[0009] The optical processing system 1 according to this embodiment includes an optical device 10 that acquires an image Ii having directional information of light from an object S as hue information for each pixel, and an optical processing device 100 that processes the image Ii acquired by the optical device 10. The object S here includes solids, liquids, gels, and the like.

[0010] In this embodiment, the image Ii acquired by the optical device 10 is mainly described as being processed by the optical processing device 100, but the image Ii is not limited to the image acquired by the optical device 10 according to this embodiment, and may be any image acquired by an optical device that acquires "an image Ii having, for each pixel, directional information of light from an object as hue information." In other words, the optical processing device 100 of the optical processing system 1 according to this embodiment can process the image Ii input to the optical processing device 100 and obtain an output (depth information for the image Ii) even without the optical device 10, as long as there is an "image Ii having, for each pixel, directional information of light from an object as hue information."

[0011] 2 is a diagram showing an example of the configuration of the optical processing device 100. The optical processing device 100 is, for example, a computer. The optical processing device 100 includes, for example, a processor (processing unit) 101, a storage medium 102, a user interface 103, and a communication module 104. The processor 101, the storage medium 102, the user interface 103, and the communication module 104 are connected to each other via a bus 105.

[0012] The processor 101 includes any one of a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a microcomputer, a field programmable gate array (FPGA), and a digital signal processor (DSP), etc. The storage medium 102 may include an auxiliary storage device in addition to a main storage device such as a memory.

[0013] The primary storage device is a non-transient storage medium. The primary storage device is, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) that can be written to and read from at any time, or a non-volatile memory such as a read only memory (ROM). These non-volatile memories may also be used in combination. The auxiliary storage device is a tangible storage medium. The auxiliary storage device is a combination of the aforementioned non-volatile memory and a volatile memory such as a random access memory (RAM). In the optical processing device 100, only one processor 101 and one storage medium 102 may be provided, or multiple processors 101 and one storage medium 102 may be provided.

[0014] In the optical processing device 100, the processor 101 performs processing by executing a program or the like stored in a storage medium 102. In addition, in the optical processing device 100, the program executed by the processor 101 may be stored in a computer (server) connected via a network such as the Internet or a server in a cloud environment. In this case, the processor 101 downloads the program via the network.

[0015] In the user interface 103, various operations are input by the user of the optical processing device 100, and information to be notified to the user is displayed. The user interface 103 may be a display unit such as a display, or an input unit such as a touch panel or a keyboard. Note that a device connected to the optical processing device 100 may be used as the input unit, or an input unit of another processing device capable of communicating via a network may be used.

[0016] FIG. 3 shows a basic processing flow of optical processing using the optical processing device 100 according to this embodiment.

[0017] First, the optical processing device 100 acquires an image Ii (an image including hue information C) having light direction information from an object as hue information for each pixel. Here, for example, as a method (optical device 10) of capturing an image using light direction information as hue information, there is Non-Patent Document 1 related to the optical device 10 shown in FIG. 5 according to the second embodiment described later, or Non-Patent Document 2 related to the optical device 10 shown in FIG. 12 and FIG. 13 according to the third embodiment described later. The hue information image Ii has hue information for two independent parameters (u, v) that indicate pixel positions.

[0018] The hue information may be, for example, H (Hue) out of the three components H (Hue), S (Saturation), and V (Value) in the HSV color coordinate space. However, it is not limited to this, and may also be the three components R (Red), G (Green), and B (Blue) in the RGB color coordinate space. In other words, the hue information may be any quantity (scalar quantity, vector quantity, or tensor quantity) that can express the hue. In the following, this quantity is referred to as C(u, v).

[0019] The light direction information may be, for example, the angle of the light ray with respect to a specific axis. Alternatively, it may be two-dimensional polar coordinates that represent the direction of the light ray. Alternatively, it may be generalized quadratic coordinates that represent the direction of the light ray in curved coordinates. In other words, the light direction information may be any quantity (scalar quantity, vector quantity, tensor quantity) that represents the direction of light. In the following, this quantity is called θ(u, v).

[0020] According to Non-Patent Document 1 related to the optical device 10 shown in FIG. 5 according to the second embodiment described later, or Non-Patent Document 2 related to the optical device 10 shown in FIG. 12 and FIG. 13 according to the third embodiment described later, the light direction information θ(u, v) can be associated with the hue information C(u, v). In other words, there exists a mapping function fc that maps the light direction information θ to the hue information C. In this case, the hue information C is calculated by using the light direction information θ as follows: C(u, v) = fc(θ(u, v)) It can be expressed as:

[0021] The optical processing device 100 of this embodiment includes an acquisition unit 111 that acquires a hue information image Ii by appropriately using a processor 101, a storage medium 102, a user interface 103, a communication module 104, etc., an image calculation unit 112 that performs calculations on the acquired hue information image Ii, and an information output unit 113 that outputs depth information (an image Io having depth information).

[0022] In the image calculation unit 112 of the optical processing device 100, at least a part of the information obtained from the hue information image Ii is input to an artificial neural network, and an output is obtained from the artificial neural network. An artificial neural network is a mathematical model consisting of a large number of interconnected "neurons", each of which processes its input using an "activation function" to generate an output. This network is known for its ability to easily generate even complex functions.

[0023] The optical processing device 100 may have an algorithm of an artificial neural network itself and obtain an appropriate output using the algorithm, or may connect to an appropriate server or the like via, for example, the user interface 103, the communication module 104, etc. of the optical processing device 100 and obtain an appropriate output using the algorithm of the artificial neural network. That is, the optical processing device 100 according to the present embodiment may output a solution using the algorithm of the artificial neural network of the optical processing device 100, or may connect to a server or the like different from the optical processing device 100 and output the image Ii to the server via, for example, the user interface 103, the communication module 104, etc., and have the solution returned from the server.

[0024] A pixel position (u, v) can be associated with a point (object point) on the imaged object. In other words, it can be associated with two-dimensional spatial coordinates (x, y) on the imaging surface. However, the two-dimensional spatial coordinates (x, y) do not necessarily have to be spatial coordinates on the actual imaging surface. In other words, anything is acceptable as long as the pixel position can be described by parameters (x, y) with two independent degrees of freedom. For example, (x, y) may be the same as (u, v).

[0025] The depth information is defined as the coordinate z(x, y) of an object in a direction perpendicular to the two-dimensional spatial coordinate (x, y).

[0026] Next, a processing flow of the optical processing device 100 according to this embodiment will be described with reference to FIG.

[0027] The hue information C of the hue information image Ii is calculated by using the light direction information θ and the mapping function fc as follows:

number

number

[0028] The mapping function fc can be obtained in advance by measuring the relationship between the light direction information θ and the hue information C, and then assuming a function such as a polynomial for the measurement results and finding it by the least squares method, etc. However, the mapping function fc is not limited to this and can be any function that links the light direction information θ and the hue information C.

[0029] As shown in FIG. 4, the image calculation unit 112 obtains light direction information θ(u, v) for a pixel position (u, v) from the hue information image Ii using equation (2).

[0030] Based on geometric optics, the light direction information θ(u, v) is considered to necessarily include shape information of the object S. For example, if the object S is opaque and has a uniform surface property, the light direction information θ is determined by the three-dimensional surface shape of the object S. Also, if the object S is transparent and has a uniform surface property and internal property, the light direction information θ is determined by the three-dimensional surface shape and three-dimensional internal shape of the object S. Even if the surface property and internal property of the object S are not uniform, the light direction information θ is at least influenced by the three-dimensional surface shape or three-dimensional internal shape of the object S. In other words, the light direction information θ necessarily includes information on the three-dimensional surface shape or three-dimensional internal shape of the object S. Hereinafter, the three-dimensional surface shape and three-dimensional internal shape of the object S will be collectively referred to simply as the three-dimensional shape of the object S.

[0031] As described above, based on geometric optics, it is possible to derive an equation that relates the light direction information θ to the three-dimensional shape of the object S. Conventional optical processing methods for acquiring the object shape from the deflection (the change in the direction of the light relative to the incident direction) are limited to the case where the object shape can be clearly formulated in a mathematical formula as a function of the deflection, that is, the object shape can be expressed by an analytical solution of the deflection. In many cases, the equation (function) becomes a nonlinear partial differential equation or an equation of a complex form, and it is difficult to find an analytical solution to the equation. However, by using an artificial neural network, it is possible to solve nonlinear partial differential equations and equations of a complex form (Non-Patent Document 4). This is due to the fact that the artificial neural network has a universal approximation theorem. In other words, the artificial neural network can approximate any smooth function with high accuracy. This property makes it possible to obtain complex solutions governed by nonlinear partial differential equations, which was previously difficult.

[0032] The pixel position (u, v) can be associated with the position of an object point OP on the imaged object S, and can be expressed by the two-dimensional spatial coordinates (x, y) of the object point OP. However, this is not limited to the above, and the two-dimensional spatial coordinates (x, y) can be anything as long as they can be associated with the pixel position (u, v).

[0033] The three-dimensional shape of the object S is equivalent to the depth information z(x, y). In this embodiment, the image calculation unit 112 uses an equation that relates the light direction information θ(u, v) and the three-dimensional shape z(x, y) of the object S, which is derived based on geometric optics, and finds a solution to the equation using an artificial neural network. Then, the three-dimensional shape z(x, y) of the object S is output from the light direction information θ(u, v).

[0034] Through the above processing, an image Io having depth information z(x, y) of the object S can be acquired from the image Ii. In this embodiment, an artificial neural network is used in the image calculation unit 112. As a result, even if the equation relating the light direction information θ(u, v) and the three-dimensional shape z(x, y) of the object S is a nonlinear partial differential equation or an equation of a complex form, a universal approximation theorem holds in the artificial neural network, so that the embodiment has the effect of being able to find a solution.

[0035] Therefore, according to this embodiment, the optical processing method includes inputting an image Ii having directional information θ of light from an object S as hue information C for each pixel to an artificial neural network, and obtaining depth information z (an image Io having depth information z) for the image Ii of the object S based on the result output by the artificial neural network. According to this embodiment, the optical processing device 100 includes a processing unit (processor) 101 that inputs an image Ii having directional information θ of light from an object S as hue information C for each pixel to an artificial neural network, and acquires depth information z (an image Io having the same) of the object S with respect to the image Ii based on the results output by the artificial neural network. According to the present embodiment, the optical processing system 1 includes such an optical processing device 100 and an imaging unit 10a that acquires an image Ii. The imaging unit 10a includes an imaging optical element 12, an image sensor 16, and a wavelength selection unit 14 that causes the image sensor 16 to acquire directional information θ of light from the object S as hue information C. According to this embodiment, the optical processing program causes a computer (processor) 101 to input an image Ii having directional information θ of light from an object S as hue information C for each pixel to an artificial neural network, and to obtain depth information z (an image Io having the same) for the image Ii of the object S based on the results output by the artificial neural network. By using such an optical processing method, optical processing device 100, and optical processing program, it is possible to acquire object information (image Io) including the three-dimensional shape z(x, y) of the object even when the object shape cannot be expressed by an analytical solution of the deflection (change in the direction of light relative to the incident direction). Note that the optical processing method, optical processing device 100, optical processing system 1, and optical processing program according to this embodiment can also be used when the object shape can be clearly formulated by a mathematical formula as a function of deflection and the object shape can be expressed by an analytical solution of the deflection.

[0036] Second embodiment The optical processing system 1 according to this embodiment will be described below with reference to FIG.

[0037] 5 shows a cross-sectional view of the optical device 10 of the optical processing system 1 according to this embodiment. The optical device 10 has an imaging means and a processing means. The optical processing device 100 of the optical processing system 1 can be the same as the optical processing device 100 described in the first embodiment.

[0038] The imaging section 10 a of the optical device 10 includes an imaging optical element 12 , a wavelength selection section 14 , and an image sensor 16 .

[0039] The imaging optical element 12 is, for example, an imaging lens. In FIG. 5, the imaging lens is depicted as a single lens, but may be a lens assembly composed of a plurality of lenses. Alternatively, the imaging optical element 12 may be a concave mirror, a convex mirror, or a combination of them. In other words, the imaging optical element 12 may be any optical element that has a function of collecting a group of rays emitted from a point on the object S, that is, an object point OP, at a conjugate image point. The imaging optical element 12 is a process in which a group of rays emitted from an object point OP on the surface of the object S is collected (concentrated) at an image point IP. Alternatively, it is also called that the object point OP is moved to an image point (a conjugate point of the object point) IP. The collection plane of the conjugate points to which the group of rays emitted from a sufficiently distant object point OP is moved by the imaging optical element 12 is called the focal plane f of the imaging optical element 12. In addition, a line perpendicular to the focal plane f and passing through the center of the imaging optical element 12 is called the optical axis Oa. In this case, the image point IP conjugate to the object point OP transferred by the light ray is called a focal point. In this embodiment, the imaging optical element 12 is simply called a lens.

[0040] In the cross section shown in Fig. 5, the optical axis Oa coincides with the z-axis. The optical axis Oa is included in the cross section shown in Fig. 5. In this embodiment, the light direction information is the angle θ between the light ray and the optical axis Oa.

[0041] The image sensor 16 has at least one pixel, and each pixel can receive light of at least two different wavelengths, that is, light of a first wavelength and light of a second wavelength. A surface including the image sensor 16 is defined as an image surface of the imaging optical element 12. The image sensor 16 may be an area sensor or a line sensor. An area sensor is a sensor in which pixels are arranged in an area on the same surface. A line sensor is a sensor in which pixels are arranged in a line. The image sensor 16 may have three color channels of R, G, and B for each pixel. In other words, the image sensor 16 may receive light at independent color channels. However, each color channel does not need to be completely independent, and each color channel may have a slight sensitivity to the same wavelength.

[0042] The wavelength selection unit 14 includes, for example, a first wavelength selection region 21 and a second wavelength selection region 22. In this embodiment, the wavelength selection unit 14 is configured as concentric circles with the first wavelength selection region 21 on the inside and the second wavelength selection region 22 on the outside.

[0043] However, the shape of the wavelength selection section 14 is not limited to this, and the wavelength selection section 14 may have any shape. The first wavelength selection region 21 passes light of a first wavelength spectrum and blocks light of a second wavelength spectrum different from the first wavelength spectrum. However, the first wavelength selection region 21 may be any shape as long as it passes light of a first wavelength spectrum and blocks at least a part of light of a second wavelength spectrum different from the first wavelength spectrum. Also, the second wavelength selection region 22 passes light of a second wavelength spectrum and blocks light of a first wavelength spectrum. However, the second wavelength selection region 22 may be any shape as long as it passes light of a second wavelength spectrum and blocks at least a part of light of a first wavelength spectrum. Here, passing light means transmitting or reflecting. In other words, it means not blocking. In this embodiment, the first wavelength spectrum is, for example, blue light with a wavelength of 400 nm to 500 nm. The second wavelength spectrum is red light with a wavelength of 580 nm to 650 nm. However, the wavelength spectrum is not limited to this, and any wavelength spectrum may be used as long as the first wavelength spectrum and the second wavelength spectrum are different from each other.

[0044] Moreover, the wavelength selection section 14 may further include a third wavelength selection region (not shown) on the outer periphery of the second wavelength selection region 22.

[0045] The wavelength selection unit 14 is disposed on the focal plane of the imaging lens 12, and is installed at a focal distance f from the principal surface of the imaging lens 12. The origin O of the (x, y, z) Cartesian coordinates is set within the wavelength selection unit 14. That is, the z axis passes through the center of the first wavelength selection region 21, and this is set as the origin O.

[0046] In this embodiment, illumination light L from an illumination unit (not shown) is irradiated as parallel light along the optical axis Oa of the imaging unit. To generate illumination light L, the illumination unit can use, for example, a white LED as a light source and generate parallel light using a pinhole and an illumination lens. Then, light is irradiated from the light source toward the optical axis Oa, and the parallel light is reflected by a half mirror or the like, illuminating the object S with parallel light (illumination light) along the optical axis Oa. Note that it is preferable that the ON / OFF of the light emission of the light source of the illumination unit is controlled by the optical processing device 100.

[0047] The directional distribution of reflected light from an object point OP on the surface (object surface) of an object S can be expressed by a distribution function called BRDF (Bidirectional Reflectance Distribution Function). BRDF generally changes depending on the surface condition (surface texture and surface shape) of the object S. For example, if the surface is rough, the reflected light spreads in various directions, so the BRDF has a wide distribution. In other words, reflected light is present over a wide range of angles. On the other hand, if the surface is mirror-finished, the reflected light will consist almost entirely of regular reflection components, and the BRDF will have a narrow distribution. In this way, the BRDF reflects the surface condition of the object S. In particular, the BRDF generally changes significantly due to tiny irregularities that are 10 times the wavelength of light or less.

[0048] In this embodiment, the object S is opaque, and the illumination light (parallel light) L is reflected on the object surface. However, the object S is not limited to this, and may be transparent or transmissive. In this case, the imaging unit 10a is disposed on the side of the object S where the illumination light (parallel light) L from the illumination unit passes through the object S.

[0049] Next, the operation of the optical processing system 1 including the processing flow of the optical processing device 100 according to this embodiment will be described.

[0050] Using the optical device 10 shown in FIG. 5, an image Ii including light direction information θ is acquired by the image sensor 16 through the imaging optical element 12 and the wavelength selection unit 14 having a region having at least two different transmission wavelength spectra or at least two different reflection wavelength spectra. When the optical device 10 shown in FIG. 5 acquires light direction information θ by the reflected light of the parallel illumination light L, the light ray from the object point OP passes through the first wavelength selection region 21, becomes a first wavelength spectrum, is imaged on the image sensor 16, and an image Ii is acquired. On the other hand, when the light direction information θ increases, the light ray from the object point OP passes through the second wavelength selection region 22, becomes a second wavelength spectrum, is imaged on the image sensor 16, and an image Ii is acquired. As a result, the hue information C acquired by the image sensor 16 changes depending on the magnitude of the light direction information θ. In other words, the light direction information θ from the object S can be identified by the hue information C acquired by the image sensor 16. That is, an image Ii including light direction information θ is acquired by an image sensor 16 through an imaging optical element 12 and a wavelength selection section 14 having a region having at least two different transmitted wavelength spectra or at least two different reflected wavelength spectra.

[0051] The position vector r represents the position where the light ray passing through the wavelength selection unit 14 at an angle (directional information of light) θ with respect to the optical axis Oa passes through the wavelength selection unit 14 when projected onto the wavelength selection unit 14. The projection position vector r on the two-dimensional plane is calculated based on the geometrical optical system using the focal length f and the two-dimensional angle vector θ having two components θx and θy.

number

[0052] Here, φ is the azimuth angle with respect to the x direction. The ray direction of θ can be obtained by color mapping of the one-shot BRDF by the optical device 10 having the wavelength selection section 14.

[0053] By increasing the number and types of shapes of the wavelength selection regions of the wavelength selection unit 14, the optical processing device 100 can more precisely distinguish the directional information θ of the light from the object S based on differences in hue. The amount representing the hue may be, for example, Hue, which can be immediately calculated from the image Ii acquired by the RGB image sensor. The mapping function fc indicates the correspondence relationship between Hue and the directional information θ of the light.

[0054] Figure 6 shows a schematic diagram of the relationship between the incident light and reflected light incident on the surface of an object S and the normal vector of the object surface. When the surface of an object is close to a mirror, the incident light is specularly reflected by the object surface. In other words, assuming that the surface is smooth and continuous, reflection can be approximated by specular reflection. In this case, the incident light ray and the reflected light ray are located within the plane of incidence. Therefore, the normal vector n at the object point OP determines the direction of the reflected light ray. Therefore, the unit vector e of the reflected light ray reflect Based on the law of specular reflection,

number

[0055] The height distribution of a three-dimensional surface can be described by a function z of (x, y). The coordinates of an object point are (x, y, z + z0), where z = z0 represents the reference plane and z represents the depth (height).

[0056] The normal vector n of the object point OP is z

number

[0057] The angle vector θ is calculated by using equations (3)-(5) to find the depth z.

number

number

[0058] Generally, the equation (7) cannot be solved analytically. Therefore, it has been difficult to solve this equation in the past.

[0059] Equation (7) is as follows:

number

[0060] FIG. 7 shows a process flow of the optical processing device 100 according to this embodiment.

[0061] Equation (8), being a nonlinear partial differential equation, can be solved recursively using an artificial neural network.

[0062] Since equation (8) is a nonlinear partial differential equation, it is difficult to solve analytically, but it can be solved using an equation-driven deep neural network (DNN) as follows: Equation-driven DNNs can construct analytical functions without requiring training data.

[0063] Here, the hue information image Ii is C(x, y), where (x, y) is two-dimensional spatial coordinates. However, (x, y) may be a pixel position. The pixel position and the two-dimensional spatial coordinates (x, y) can be considered equivalent. However, since the spatial coordinates of the image Ii are discrete, they are numbered with subscripts. In other words, (x i , y i) (i = 1, 2, …, M). The loss function is ζ, which can be expressed by equation (8), and is derived from the two-dimensional spatial coordinates (x, y) and light direction information θ input to the DNN.

[0064] It is important to note here that equation-driven DNN is an unsupervised learning method and does not require data to train the DNN. Moreover, the universal approximation theorem for neural networks states that DNNs can approximate any smooth function with high accuracy.

[0065] The depth (height) distribution z on the 3D surface of the object S is obtained by making the loss function ζ approach zero. This involves repeatedly reducing the loss function ζ using the backpropagation algorithm of the DNN. That is, the optical processing device 100 inputs an image Ii to the artificial neural network, that is, iteratively calculates the three-dimensional shape (image Io) of the object S output when light direction information θ is input using the artificial neural network so that it satisfies an equation (loss function ζ=0) derived based on geometric optics.

[0066] For example, in this embodiment, the DNN is composed of three hidden layers and one output layer. Each hidden layer is composed of 100 neurons. The activation function of the hidden layer is a hyperbolic tangent function, and the activation function of the output layer is scaled by a sigmoid function. The total number of grids M is, for example, 65536, and the number of grids in both the x and y directions is 256.

[0067] By the above process, the information of image Ii (x1, x2, …, x M ), (y1, y2, …, y M ), (C1, C2, …, C M ) based on (x1, x2, …, x M ), (y1, y2, …, y M ), (θ1, θ2, …, θ M ) into an artificial neural network and (z1, z2, …, z M), we iteratively approach the loss function ζ=0 using the backpropagation algorithm to satisfy the equation for the loss function ζ=0 (equation (8)), and then we obtain (z1, z2, …, z M ) to obtain the solution.

[0068] In this way, the solution to equation (8) is z1(x1, y1) to z M (x M , y M ) is obtained, and the depth information z(x, y) of the object S can be obtained.

[0069] In this embodiment, an artificial neural network is used in the image calculation unit 112. As a result, even if the equation relating the light direction information θ(u, v) and the three-dimensional shape z(x, y) of the object S is a nonlinear partial differential equation or an equation of a complex form, a universal approximation theorem holds in the artificial neural network, so that the optical processing device 100 according to this embodiment can find a solution.

[0070] In this embodiment, the wavelength selection unit 14 is concentric. This means that the directional information θ of light with respect to the optical axis Oa can be obtained over the entire imaging field of view of the image Ii. If the directional information θ of light can be obtained, the inclination angle of the object S having a surface close to a mirror surface can be calculated according to the law of regular reflection. This makes it possible to instantly obtain, for example, the contact angle of a liquid on a solid surface over the entire field of view of the image Ii. Furthermore, there is an effect that the three-dimensional shape of the liquid can also be obtained by the optical processing device 100 according to this embodiment. That is, there is an effect that the inclination angle of the object S with respect to the optical axis Oa and the three-dimensional shape of the object S can be obtained over the entire field of view of the image Ii. That is, the depth information (image Io having the depth information) according to this embodiment includes not only the depth (height) along the z axis but also, for example, the contact angle on the solid surface (the surface on which the object S is placed), and therefore includes the three-dimensional shape of the object S.

[0071] Fig. 8 shows an example of an image Ii including hue information C captured by the optical device 10 of an aluminum surface. The object shape of this image Ii cannot be expressed by an analytical solution of the deflection. The image Ii shown in Fig. 8 has hue information for each pixel.

[0072] Fig. 9 shows an example of the wavelength selecting unit 14 used in the optical device 10 to obtain the image Ii shown in Fig. 8. Note that the wavelength selecting unit 14 shown in Fig. 9 differs from the arrangement of the wavelength selecting regions 21, 22 of the wavelength selecting unit 14 shown in Fig. 5. The wavelength selecting unit 14 shown in Fig. 9 and the light direction information θ are associated with each other by the hue information of the image Ii shown in Fig. 8.

[0073] Fig. 10 shows an example of a three-dimensional image Io including depth information obtained by inputting the image Ii shown in Fig. 8 into the optical processing device 100 according to this embodiment and obtaining a solution using an artificial neural network. Fig. 11 shows the distribution of inclination angles of the angle α of the surface of the object S with respect to the horizontal plane based on the image Ii shown in Fig. 8 and the three-dimensional image Io shown in Fig. 10.

[0074] In this way, based on the image Ii (see Figure 8) obtained by the optical device 10 of the optical processing system 1 of this embodiment, object information regarding depth information such as the angle with respect to the horizontal plane at each position of the object S (see Figures 10 and 11) can be obtained.

[0075] Therefore, according to this embodiment, it is possible to provide an optical processing method, an optical processing device 100, an optical processing system 1, and an optical processing program that can acquire object information even when the object shape cannot be expressed by an analytical solution of deflection (directional change of light relative to the incident direction).

[0076] Third embodiment The optical processing system 1 according to this embodiment will be described below with reference to FIGS.

[0077] The basic parts of the optical processing system 1 according to this embodiment are the same as those of the optical processing system 1 according to the second embodiment, but the differences will be described below.

[0078] 12 and 13 are perspective views of the optical device 10 of the optical processing system 1 according to this embodiment. The imaging optical element 12 (see FIG. 5) disposed between the wavelength selection unit 14 and the object S in the optical device 10 of FIGS. 12 and 13 is omitted.

[0079] The wavelength selection section 14 of the present embodiment is striped. In other words, the wavelength selection section 14 of the optical device 10 according to the present embodiment is not concentric. The wavelength selection section 14 includes, for example, a first wavelength selection region 31, a second wavelength selection region 32, and a third wavelength selection region 33.

[0080] The first wavelength selection region 31 passes light of a first wavelength spectrum and blocks light of a second wavelength spectrum different from the first wavelength spectrum and a third wavelength spectrum different from the first wavelength spectrum. However, the first wavelength selection region 31 may be any type that passes light of the first wavelength spectrum and blocks wavelengths not included in the first wavelength spectrum. The second wavelength selection region 32 passes light of the second wavelength spectrum and blocks light of the first wavelength spectrum and the third wavelength spectrum. However, the second wavelength selection region 32 may be any type that passes light of the second wavelength spectrum and blocks wavelengths not included in the second wavelength spectrum. The third wavelength selection region 33 passes light of the third wavelength spectrum and blocks light of the first wavelength spectrum and the second wavelength spectrum. However, the third wavelength selection region 33 may be any type that passes light of the third wavelength spectrum and blocks wavelengths not included in the third wavelength spectrum. The first wavelength spectrum is, for example, blue light with wavelengths from 400 nm to 500 nm, the second wavelength spectrum is, for example, red light with wavelengths from 580 nm to 650 nm, and the third wavelength spectrum is, for example, green light with wavelengths from 480 nm to 600 nm.

[0081] 12 and 13 have different arrangements of the wavelength selection section 14. The wavelength selection section 14 in FIG. 13 is rotated 90° around the optical axis Oa with respect to the wavelength selection section 14 shown in FIG. 12. In FIG. 12, the direction in which the wavelength selection regions 31, 32, and 33 change is the x-axis direction. That is, each of the wavelength selection regions 31, 32, and 33 shown in FIG. 12 is, for example, a rectangular shape that is long in the y-axis direction. On the other hand, in FIG. 13, the direction in which the wavelength selection regions 31, 32, and 33 change is the y-axis direction. That is, each of the wavelength selection regions 31, 32, and 33 shown in FIG. 13 is, for example, a rectangular shape that is long in the x-axis direction.

[0082] The optical device 10 includes an actuator such as a motor that rotates the wavelength selection unit 14 around the z-axis (optical axis Oa) to a position shown in Fig. 12 and a position shown in Fig. 13 and stops the rotation. Such an actuator is preferably controlled by the optical processing device 100.

[0083] The operation of the optical processing system 1 according to this embodiment will now be described.

[0084] In this embodiment, the optical processing system 1 causes the wavelength selection unit 14 of the optical device 10 to acquire at least one image for each of an object point OP of the same object S when the wavelength selection unit 14 is arranged as shown in Fig. 12 and when the wavelength selection unit 14 is arranged as shown in Fig. 13. That is, at least one hue information image Ii is acquired in each arrangement, and a total of two images Ii are acquired.

[0085] In the case of Fig. 12, the hue information image Ii acquired by the optical device 10 corresponds to the angle θx that the light ray makes with the z-axis (optical axis) when the light ray is projected onto the xz plane. On the other hand, in the case of Fig. 13, the hue information image Ii acquired by the optical device 10 corresponds to the angle θy that the light ray makes with the z-axis (optical axis) when the light ray is projected onto the yz plane. In this case, if the depth information is z and the object surface is approximately a mirror surface, then according to geometric optics,

number

[0086] The artificial neural network can also simultaneously solve these two equations expressed by equation (9). In other words, the artificial neural network can iteratively calculate the solution of the two equations in equation (9) by satisfying the loss function ζx,y=0. In this case, two hue information images Ii are input to the artificial neural network. In other words, the number of hue information images Ii input to the artificial neural network is not limited to one, but may be two or more.

[0087] In this way, by simultaneously inputting the two images Ii into the artificial neural network and calculating the depth information z, there is an advantage that the error caused by the optical device 10 when the image Ii is acquired can be mitigated. In other words, for example, there is an advantage that the error is reduced by using the image with a small error as well as the image with a small error out of the two images Ii, since the errors are averaged, rather than using only the image with a large error out of the two images Ii.

[0088] In other words, by increasing the number of hue information images Ii to be input, (z1, z2, …, z M ) (the three-dimensional shape (image Io) of object S) is reduced.

[0089] This embodiment has an effect of functioning as an optical processing system 1 that acquires depth information z of an object S. Therefore, according to this embodiment, even if the object shape cannot be expressed by an analytical solution of deflection (a change in the direction of light relative to the incident direction), it is possible to provide an optical processing method, an optical processing device 100, an optical processing system 1, and an optical processing program that are capable of acquiring object information.

[0090] (Fourth embodiment) The optical processing system 1 according to this embodiment will be described below with reference to FIG.

[0091] The basic parts of the optical inspection system 1 according to this embodiment are the same as those of the optical processing systems 1 according to the second and third embodiments, but the differences will be described below.

[0092] 14 shows a perspective view of the optical device 10 of the optical processing system 1 according to this embodiment and an optical processing flow. In the optical device 10 of FIG. 14, the imaging optical element 12 (see FIG. 5) disposed between the wavelength selection unit 14 and the object S is omitted.

[0093] The wavelength selection section 14 of this embodiment has anisotropy in the azimuth direction. In other words, the wavelength selection section 14 is not concentric. The wavelength selection section 14 includes a first wavelength selection region 41, a second wavelength selection region 42, and a third wavelength selection region 43. Although the outer shape of the wavelength selection section 14 is triangular, the outer shape may be any shape, such as a circle or a rectangle.

[0094] The first wavelength selection region 41 passes light of a first wavelength spectrum and blocks light of a second wavelength spectrum different from the first wavelength spectrum and light of a third wavelength spectrum different from the first wavelength spectrum. However, the first wavelength selection region 41 may be any type that passes light of the first wavelength spectrum and blocks light of wavelengths not included in the first wavelength spectrum. The second wavelength selection region 42 passes light of the second wavelength spectrum and blocks light of the first wavelength spectrum and the third wavelength spectrum. However, the second wavelength selection region 42 may be any type that passes light of the second wavelength spectrum and blocks light of wavelengths not included in the second wavelength spectrum. The third wavelength selection region 43 passes light of the third wavelength spectrum and blocks light of the first wavelength spectrum and the second wavelength spectrum. However, the third wavelength selection region 43 may be any type that passes light of the third wavelength spectrum and blocks light of wavelengths not included in the third wavelength spectrum. The first wavelength spectrum is, for example, blue light with wavelengths from 400 nm to 500 nm, the second wavelength spectrum is, for example, red light with wavelengths from 580 nm to 650 nm, and the third wavelength spectrum is, for example, green light with wavelengths from 480 nm to 600 nm.

[0095] The z-axis (optical axis Oa) passes through, for example, the center of the wavelength selection unit 14. The wavelength selection unit 14 according to this embodiment preferably has three wavelength selection regions 41, 42, and 43, with a central angle of 120°. Note that even when the wavelength selection unit 14 has three wavelength selection regions 41, 42, and 43, the central angles may all be different.

[0096] Furthermore, the central angle is changed depending on the number of wavelength selection regions in the wavelength selection section 14.

[0097] The optical processing system 1 according to this embodiment includes an optical processing device 100, which is the same as the optical processing device 100 described in the first embodiment.

[0098] Furthermore, the optical processing device 100 according to this embodiment includes an image transfer unit 111a that transfers the image Ii acquired by the image sensor 16. This image transfer unit 111a can be used in the same manner as the acquisition unit 111 described in the first embodiment.

[0099] In addition, the optical processing device 100 may also include a GUI (Graphical User Interface) or the like, but this is omitted from the illustration in the optical processing system 1 shown in FIG.

[0100] The operation of the optical processing system 1 according to this embodiment will now be described. As described in the third embodiment, in this embodiment, the optical processing system 1 acquires at least one hue information image Ii for an object point OP of the same object S when the wavelength selection unit 14 of the optical device 10 is rotated 60° and 180° around the optical axis Oa with respect to the arrangement shown in Fig. 14, respectively. That is, at least one hue information image Ii is acquired in each arrangement, and a total of three images Ii are acquired.

[0101] By providing such anisotropy to the wavelength selection unit 14, it is possible to obtain directional information θ of light related to the azimuth angle direction based on the image Ii. In addition, by obtaining an image while rotating it around the optical axis Oa, it is possible to obtain more information related to the azimuth angle direction.

[0102] For example, by rotating the wavelength selection unit 14 around the optical axis Oa in accordance with the central angle of each of the wavelength selection regions 41, 42, and 43, obtaining an image Ii each time, and inputting, for example, three images Ii simultaneously into an artificial neural network to obtain depth information z, it is possible to mitigate errors caused by the optical device 10 when obtaining the image Ii. In other words, for example, there is an effect that errors are averaged and therefore errors are reduced by outputting using images with small errors as well, rather than outputting using only images with large errors out of the three images Ii.

[0103] This embodiment has an effect of functioning as an optical processing system 1 that acquires depth information z of an object S. Therefore, according to this embodiment, even if the object shape cannot be expressed by an analytical solution of deflection (a change in the direction of light relative to the incident direction), it is possible to provide an optical processing method, an optical processing device 100, an optical processing system 1, and an optical processing program that are capable of acquiring object information.

[0104] According to at least one of the above-described embodiments of the optical processing method, optical processing device 100, optical processing system 1, and optical processing program, object information can be obtained even when the object shape cannot be expressed by an analytical solution of deflection (change in direction of light relative to the incident direction).

[0105] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0106] 1...optical processing system, 10...optical device, 12...imaging optical element, 14...wavelength selection section, 16...image sensor, 100...optical processing device, 101...processor, 102...storage medium, 103...user interface, 104...communication module, 105...bus, 111...acquisition section, 112...image calculation section, 113...information output section.

Claims

1. An image having the directional information of light from an object as hue information for each pixel is input to an artificial neural network. obtaining depth information of the object relative to the image based on the output of the artificial neural network; An optical processing method comprising:

2. and iteratively calculating the three-dimensional shape of the object output when the light direction information is input using the artificial neural network so that the three-dimensional shape satisfies an equation derived based on geometric optics. The optical processing method according to claim 1 .

3. The light direction information is based on a specific axis and is the angle of a light ray with respect to the axis. The optical processing method according to claim 1 or 2.

4. The image including the directional information of the light is acquired by an image sensor through an imaging optical element and a wavelength selection unit having a region having at least two different transmission wavelength spectra or at least two different reflection wavelength spectra. The optical processing method according to claim 1 or 2.

5. A processing unit is provided for inputting an image having light direction information from an object as hue information for each pixel into an artificial neural network, and acquiring depth information of the object relative to the image based on a result output by the artificial neural network. Optical processing equipment.

6. The optical processing device according to claim 5 ; An imaging unit for acquiring the image; Equipped with The imaging unit includes an imaging optical element, an image sensor, and a wavelength selection unit that causes the image sensor to acquire directional information of light from the object as hue information. Optical processing system.

7. The wavelength selection unit is disposed on a focal plane of the imaging optical element.

7. The optical processing system of claim 6.

8. The wavelength selection portion is concentric.

8. An optical processing system according to claim 6 or claim 7.

9. The wavelength selection portion is striped.

8. An optical processing system according to claim 6 or claim 7.

10. An image having the directional information of light from an object as hue information for each pixel is input to an artificial neural network. obtaining depth information of the object relative to the image based on the output of the artificial neural network; An optical processing program that causes a computer to execute the above.

Images