Image analysis method and image analysis device

Abstract

To analyze the correspondence between a reference pattern image and a photographed image generated from a photographed object on which a pattern light is projected.SOLUTION: A reference image includes a first set of longitudinally extending elements and a second set of laterally extending elements, representing that either the first set of elements or the second set of elements includes intermittently extending elements. A method includes a step of identifying each node composed of the first set of elements and the second set of elements in the captured image, a step of classifying the nodes based on their relative positions to adjacent nodes to each of the nodes and assigning a classification value to each of the nodes, a step of generating a lattice graph from the classified nodes, a step of determining, for each node in the lattice graph, the adjacency to adjacent nodes, and a step of associating the nodes in the lattice graph with the nodes in the reference image based on the classification value and the adjacency.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an image analysis method and an image analysis device, and in particular to an image analysis method and an image analysis device that analyze the correspondence between a reference pattern image and a captured image generated from a subject onto which pattern light is projected. [Background technology]

[0002] There are technologies that can create a sense of depth in two-dimensional images (i.e., construct a three-dimensional image) by capturing an object (subject) using an imaging device (camera). For example, stereo imaging is a typical technology for creating three-dimensional images.

[0003] Stereo imaging methods are generally classified into passive stereo and active stereo. In active stereo, a light projector installed separately from the imaging device projects light onto the subject, and the imaging device then captures the subject. In active stereo, the light from the projector is reflected from the subject and reaches the imaging device. This means that the optical path of the light from the projector is shifted according to a distance corresponding to the depth of the subject. Therefore, active stereo uses this optical path shift to calculate the distance between the imaging device and the subject based on triangulation and express a three-dimensional image (depth) based on this distance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-300277 Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned active stereo method, in order to calculate the distance corresponding to the deviation of the optical path, it is necessary to associate the original position of the light to be projected onto the object on a two-dimensional plane of the object with the position of the light actually projected. Among the active stereo methods, there is a pattern light projection method, which projects light that forms a specific pattern (such as a grid) from a projector. The pattern light projection method can reduce the number of times the object is photographed, compared to, for example, a method that projects spot light.

[0006] However, the pattern light projection method requires that the original position of the pattern light be associated with the actual projected position for each of the multiple gratings, making this association difficult. This is because the shape of the pattern light reflected from the subject is distorted depending on the depth of the subject, making it difficult to determine which grating of the pattern light to be projected corresponds to which grating of the actually projected pattern light.

[0007] There is a technology that solves the above-mentioned problem by considering the epipolar geometry between the projection device and the imaging device. Considering the epipolar geometry requires that the positional relationship between the projection device and the imaging device be known. For example, when the imaging device is incorporated into an endoscope and an image of an object inside the human body is captured, the endoscope enters the human body, making it impossible to know the positional relationship between the projection device and the imaging device. In such a case, it is impossible to consider the epipolar geometry, making it difficult to perform the above-mentioned correspondence.

[0008] Patent Document 1 discloses a technique for achieving three-dimensional reconstruction by using a simple grid pattern made up of a large number of straight lines in which only the vertical and horizontal directions can be distinguished, and using the intersections of the grid pattern as feature points. The technique disclosed in Patent Document 1 can easily achieve three-dimensional reconstruction by using a simple pattern, but it does not solve the above-mentioned problems. [Means for solving the problem]

[0009] In one embodiment, a computer-implemented method analyzes correspondences between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, the reference image including a first set of elements extending vertically in a two-dimensional plane and a second set of elements extending horizontally in the two-dimensional plane, and the reference image represents at least one of the first set of elements and the second set of elements including intermittently extending elements, the method comprising: identifying, in the captured image, respective nodes composed of the first set of elements and the second set of elements; classifying each of the nodes based on its relative position with respect to adjacent nodes and assigning a classification value; generating a lattice graph from the classified nodes; and matching nodes in the captured image with nodes in the reference image by inputting the classification values ​​and the lattice graph into a graph convolutional network (GCN), the GCN being configured to learn the classification values ​​for the classified nodes from the reference image and the lattice graph generated from the reference image.

[0010] Another embodiment of the method is a computer-implemented method for analyzing a correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, the reference image including a first set of elements extending vertically or horizontally in a two-dimensional plane and a second set of elements extending intermittently in the same direction as the first set of elements, the reference image representing each of the second set of elements being arranged at random intervals in the same direction, the method including the steps of: identifying, in the captured image, each node composed of the first set of elements and the second set of elements; classifying each of the nodes based on a relative angle with respect to adjacent nodes and assigning a classification value to the node; generating a lattice graph from the classified nodes; and associating the nodes in the captured image with nodes in the reference image by inputting the classification values ​​and the lattice graph into a graph convolutional network (GCN), the GCN being configured to learn the classification values ​​for the classified nodes from the reference image and the lattice graph generated from the reference image. [Effects of the Invention]

[0011] The image analysis method and image analysis device according to the embodiment facilitate associating an original pattern with a pattern contained in an image of a photographed subject. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image analysis system. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of a computer device. [Figure 3] FIG. 2 is a diagram showing an example of a reference image according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing another example of the reference image according to the first embodiment. [Figure 5] 4 is a flowchart illustrating an example of processing executed by the image analysis system according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of classifying nodes in an image. [Figure 7] FIG. 10 is a diagram showing a region with a marker at the center. [Figure 8] FIG. 1 is a block diagram illustrating an example of the configuration of a GCN. [Figure 9] FIG. 10 is a diagram illustrating an operation that takes into account feature vectors, relationships with adjacent nodes, and weight values. [Figure 10] 10 is a flowchart illustrating an example of processing executed by an image analysis system according to a second embodiment. [Figure 11] 10 shows an example of a process for calculating the inner product of feature vectors of a reference image and a captured image according to the second embodiment. [Figure 12] FIG. 11 is a diagram showing an example of a reference image according to the third embodiment. [Figure 13] FIG. 11 is a diagram showing an example of processing for determining the phase of a grid (the relative positional relationship between a repeating grid and each pixel expressed as a rotation angle between 0 and less than 2π) from a captured image according to the third embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of a connection based on a proximity relationship according to the third embodiment. [Figure 15] FIG. 11 is a diagram showing an example of classifying nodes in an image according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] An image analysis method and an image analysis device according to an embodiment will be described in detail below with reference to the accompanying drawings. In this embodiment, the image analysis method and the image analysis device are implemented in an image analysis system. In this embodiment, an object onto which light from a projector is projected and photographed by a camera is referred to as a "photographed object." The photographed object includes any object having depth, such as a human, an animal, an object, or a part thereof.

[0014] An image analysis system according to an embodiment will be described below. The image analysis system is applied to an example of capturing an image of an object inside a human body. Based on an active stereo method, the image analysis system analyzes the correspondence between a pattern image (reference image) corresponding to a pattern light projected onto the object and a captured image generated by capturing an image of the object onto which the pattern light is projected.

[0015] First Embodiment First, an example of the configuration of an image analysis system 100 will be described with reference to Fig. 1. In this embodiment, the image analysis system 100 includes a computer device 1 and an endoscope 2. The computer device 1 is mutually coupled to the endoscope 2 via a bus, a network, or the like.

[0016] The computing device 1 is any information processing device that includes at least a calculation function. The computing device 1 generates an image based on an imaging signal received from an imaging device (described later), and analyzes the correspondence between a reference image corresponding to a pattern light projected by a light projecting device and a captured image generated by capturing an image of a target onto which the pattern light is projected.

[0017] The endoscope 2 includes an imaging device 21 (camera) and a light projecting device 22 built into the tip (head). In Fig. 1, the imaging device 21, the light projecting device 22, and the endoscope head 23 are shown enlarged inside an ellipse O2 corresponding to the ellipse O1 surrounding the tip of the endoscope 2. The endoscope head 23 is inserted inside the human body, the light projecting device 22 projects pattern light PL onto the subject, and the imaging device 21 photographs the subject.

[0018] The imaging device 21 includes one or more CCD image sensors or CMOS image sensors, captures an image of a subject, and transmits a captured image signal to the computer device 1. It is desirable that the imaging device 21 has an array of as many image sensors as possible to generate an image with a high pixel count.

[0019] The light projecting device 22 includes a diffractive optical element (DOE) 22a, a lens 22b, and an optical fiber 22c. In FIG. 1, an enlarged view of the diffractive optical element 22a, the lens 22b, and the optical fiber 22c is shown inside a rectangle R2 corresponding to a rectangle R1 that surrounds the light projecting device 22.

[0020] The diffractive optical element 22a diffracts the pattern light PL to be projected. The lens 22b is implemented, for example, by a gradient index (GRIN) lens. By implementing a gradient index lens, the refractive index of light can be changed in the radial direction, making it possible to reduce the size of the light projecting device 22. Light from a light source (not shown) propagates through the optical fiber 22c and radiates the pattern light PL via the lens 22b and the diffractive optical element 22a. The pattern light PL will be described in detail later.

[0021] In this embodiment, a configuration is adopted in which the imaging device 21 and the light projecting device 22 are incorporated into the endoscope 2 to capture an image of an object inside the human body, but this embodiment is not limited to such a configuration. For example, to capture an arbitrary object, an imaging device that captures the object and a light projecting device that projects pattern light onto the object may exist independently. In other words, this embodiment adopts a configuration of an active stereo system that includes at least an imaging device that captures the object and a light projecting device that projects pattern light onto the object.

[0022] Next, detailed components of the computer device 1 will be described with reference to Fig. 2. The computer device 1 includes a control device 11, a memory 12, a storage device 13, a communication device 14, an input device 15, and an output device 16. The memory 12, the storage device 13, the communication device 14, the input device 15, and the output device 16 are each coupled to the control device 11 via an internal bus and are controlled by the control device 11.

[0023] The control device 11 is also referred to as a processor, and includes a central processing unit (CPU), a graphics processing unit (GPU), etc. The control device 11 generates an image based on the imaging signal received from the imaging device 21, and performs calculations such as analyzing the correspondence between the generated image and a reference image corresponding to the pattern light PL projected by the light projecting device 22 based on the pattern in the image.

[0024] The memory 12 is a volatile data storage device that stores computer-executable instructions processed by the control device 11, data after arithmetic processing of the instructions, etc. The memory 12 may be implemented as a RAM (random access memory) (e.g., SRAM (static RAM) and DRAM (dynamic RAM)).

[0025] The storage device 13 is a non-volatile data storage device that stores programs including the above-mentioned computer-executable instructions. The storage device 13 may be implemented by a non-volatile semiconductor memory such as a read-only memory (ROM), a magnetic storage device (such as a hard disk drive), an optical disk, or the like. Note that data such as programs may be stored in a network-attached storage (NAS) and / or a storage area network (SAN) in addition to or instead of the storage device 13.

[0026] The communication device 14 communicates with external devices coupled to the computer device 1, such as by receiving an imaging signal from the endoscope 2 (imaging device 21) and transmitting a signal corresponding to a reference image to the endoscope 2 (light projector 22).

[0027] The input device 15 receives input from a user and transmits the received input to the control device 11. The input device 15 is implemented by, for example, a mouse, a touchpad, a keyboard, a trackball, or the like.

[0028] The output device 16 outputs the results of calculations performed by the control device 11 (for example, displays the reference image generated by the control device 11). The output device 16 is implemented by, for example, a display (liquid crystal, CRT, etc.).

[0029] In this embodiment, the input device 15 and the output device 16 are incorporated into the computer device 1, but the present invention is not limited to such a configuration. Either or both of the input device 15 and the output device 16 may be configured as devices independent of the computer device 1.

[0030] Next, the reference image RI constituting the pattern light PL emitted by the light projecting device 22 will be described with reference to Fig. 3. As described above, the image analysis system 100 projects the pattern light PL onto the subject. The reference image RI is an image that represents a predetermined pattern and is stored in the computer device 1 or the endoscope 2, and the light projecting device 22 emits the pattern light PL based on the reference image RI.

[0031] The reference image RI includes a set of elements (a set of multiple first elements) SE1 extending vertically in a two-dimensional plane, and a set of elements (a set of multiple second elements) SE2 extending horizontally in the two-dimensional plane. In the example shown in FIG. 3, the set of first elements SE1 is a set of straight lines extending vertically in the two-dimensional plane, each of which is a first element E1a to E1n (n is any integer equal to or greater than 2). The set of second elements SE2 is a set of straight lines extending horizontally in the two-dimensional plane, each of which is a second element E2a to E2n (n is any integer equal to or greater than 2). Each first element in the set of first elements SE1 intersects with each second element in the set of second elements SE2. Note that the value of n mentioned above and the value of n mentioned below refer to any integer equal to or greater than 2, and not all of the n's mentioned are the same value.

[0032] As shown in Fig. 3, each straight line in the first element set SE1 extends continuously vertically in a two-dimensional plane within the reference image RI. Meanwhile, a portion of each straight line in the second element set SE2 extends intermittently horizontally in the two-dimensional plane within the reference image RI. Fig. 3 shows an enlarged view of a portion of the reference image RI surrounded by circle C1 within circle C2, which corresponds to circle C1 surrounding a portion of the reference image RI.

[0033] In the area within the circle C2, the first elements E1a to E1d each extend continuously in the vertical direction. The second elements E2a, E2d, and E2g extend intermittently in the horizontal direction. The second elements E2b, E2e, and E2h extend continuously in the horizontal direction. The second elements E2c, E2f, and E2i extend intermittently in the horizontal direction.

[0034] The second element E2a extends laterally to the right from the intersection I1 with the first element E1a and terminates at the intersection I2 with the first element E1b. The second element E2d extends laterally to the right from the intersection I3 with the first element E1b and terminates at the intersection I4 with the first element E1c. The second element E2g extends laterally to the right from the intersection I5 with the first element E1c and terminates at the intersection I6 with the first element E1d. The second element E2d extends from the intersection I3 at a higher position than the second element E2a, and the second element E2g extends from the intersection I5 at a lower position than the second element E2d.

[0035] The second element E2b extends laterally to the right from the intersection I7 with the first element E1a and connects to the second element E2e at the intersection I8 with the first element E1b. The second element E2e extends laterally to the right from the intersection I8 and connects to the second element E2h at the intersection I9 with the first element E1c. The second element E2h extends laterally to the right from the intersection I9 and connects to the next second element (unsigned) at the intersection I10 with the first element E1d.

[0036] The second element E2c extends laterally to the right from the intersection I11 with the first element E1a and terminates at the intersection I12 with the first element E1b. The second element E2f extends laterally to the right from the intersection I13 with the first element E1b and terminates at the intersection I14 with the first element E1c. The second element E2i extends laterally to the right from the intersection I15 with the first element E1c and terminates at the intersection I16 with the first element E1d. The second element E2f extends from the intersection I13 at a lower position than the second element E2c, and the second element E2i extends from the intersection I15 at a higher position than the second element E2f.

[0037] Furthermore, in the reference image RI, n markers of any shape are arranged at positions on predetermined intersections. In this embodiment, nine markers M1 to M9 are arranged. The markers M1 to M9 may be arranged at regular intervals on a two-dimensional plane of the reference image RI, or may be arranged at random intervals. The same applies to the reference image shown in FIG. 4, which will be described later. Each of the markers M1 to M9 is used to improve the accuracy of correspondence, and details will be described later.

[0038] Although the present embodiment employs a reference image including a set of elements that extend continuously in the vertical direction in a two-dimensional plane and a set of elements that extend intermittently in the horizontal direction in the two-dimensional plane, the reference image is not limited to such a configuration. For example, as shown in Fig. 4, the reference image RI may include a set of elements that extend intermittently in the vertical direction in a two-dimensional plane and a set of elements that extend continuously in the horizontal direction in the two-dimensional plane.

[0039] As shown in Figure 4, a portion of each straight line in the first element set SE1 extends intermittently in the vertical direction in a two-dimensional plane within the reference image RI. Meanwhile, each straight line in the second element set SE2 extends continuously in the horizontal direction in the two-dimensional plane within the reference image RI. Figure 4 shows an enlarged view of a portion of the reference image RI surrounded by circle C1 within circle C2, which corresponds to circle C1 surrounding the portion of the reference image RI.

[0040] In the region within the circle C2, the second elements E2a to E2d each extend continuously in the horizontal direction. The first elements E1a, E1d, and E1g extend intermittently in the vertical direction. The first elements E1b, E1e, and E1h extend continuously in the vertical direction. The first elements E1c, E1f, and E1i extend intermittently in the vertical direction.

[0041] The first element E1a extends downward from the intersection I1 with the second element E1a and terminates at the intersection I2 with the second element E2b. The first element E1d extends downward from the intersection I3 with the second element E2b and terminates at the intersection I4 with the second element E2c. The first element E1g extends downward from the intersection I5 with the second element E2c and terminates at the intersection I6 with the second element E2d. The first element E1d extends from the intersection I3 to a position to the left of the first element E1a, and the first element E1g extends from the intersection I5 to a position to the right of the first element E1d.

[0042] The first element E1b extends downward from the intersection I7 with the second element E2a and connects to the first element E1e at the intersection I8 with the second element E2b. The first element E1e extends downward from the intersection I8 and connects to the first element E1h at the intersection I9 with the second element E2c. The first element E1h extends downward from the intersection I9 and connects to the next first element (unsigned) at the intersection I10 with the second element E2d.

[0043] The first element E1c extends downward from an intersection I11 with the second element E2a and terminates at an intersection I12 with the second element E2b. The first element E1f extends downward from an intersection I13 with the second element E2b and terminates at an intersection I14 with the second element E2c. The first element E1i extends downward from an intersection I15 with the second element E2c and terminates at an intersection I16 with the second element E2d. The first element E1f extends from the intersection I13 to the right of the first element E1c, and the first element E1i extends from the intersection I15 to the left of the first element E1f.

[0044] 3 and 4 can be said to include a set of straight line elements extending vertically in a two-dimensional plane, and a set of straight line elements extending horizontally in the two-dimensional plane, each of which intersects with the set of straight line elements extending vertically. Furthermore, the above patterns can be said to include elements in which at least one of the set of straight line elements extending vertically and the set of straight line elements extending horizontally extends intermittently.

[0045] An element extending intermittently in a two-dimensional plane means that the element intersects with the same element of the other intersecting elements at multiple positions, and the positions of the intersections are different in the two-dimensional plane. In the pattern shown in Figure 3, a second element extending intermittently in the horizontal direction in a two-dimensional plane means that a second element (e.g., second elements E2a and E2d) intersects with the same first element (e.g., first element E1b) at multiple positions (e.g., intersections I2 and I3), and the positions of the intersections are different in the two-dimensional plane. In the pattern shown in Figure 4, a first element extending intermittently in the vertical direction in a two-dimensional plane means that a first element (e.g., first elements E1a and E1d) intersects with the same second element (e.g., second element E2b) at multiple positions (e.g., intersections I2 and I3), and the positions of the intersections are different in the two-dimensional plane.

[0046] In this embodiment, the reference image RI includes a set of straight lines extending vertically in a two-dimensional plane and a set of straight lines extending horizontally in the two-dimensional plane, but is not limited to such a configuration. For example, the reference image RI may include a set of elements having an arbitrary shape, such as a set of curved lines extending vertically in a two-dimensional plane, and a set of elements having an arbitrary shape, such as a set of curved lines extending horizontally in the two-dimensional plane.

[0047] Next, an example of processing executed by the image analysis system 100 will be described with reference to the flowchart shown in FIG. 5. As described above, in this embodiment, the correspondence between a reference image corresponding to a pattern light projected onto a subject and a captured image generated by capturing an image of the subject onto which the pattern light is projected is analyzed. Specifically, the correspondence between the reference image shown in FIG. 3 or 4 and a captured image generated by capturing an image of the subject onto which pattern light corresponding to the reference image is projected is analyzed. The correspondence is analyzed using a neural network.

[0048] When a pattern light is projected onto a subject, the shape of the pattern light reflected from the subject is distorted depending on the depth, and the shape of the pattern recognized in the generated captured image is also distorted accordingly. Conventional image processing may not always be able to accurately recognize a pattern from a captured image containing such a distorted pattern. In this embodiment, a neural network learns the nodes that make up the pattern in the reference image, and, based on information about the learned nodes, extracts the nodes that make up the pattern in the captured image by associating them with the nodes in the reference image.

[0049] In this embodiment, it is assumed that any one of the reference images RI shown in Fig. 3 is generated in advance and stored in the storage device 13 of the computer device 1. The reference image RI includes the above-mentioned markers M1 to M9.

[0050] First, the light projecting device 22 of the endoscope 2 projects pattern light PL corresponding to the reference image RI onto the subject to be photographed (step S501). The reference image RI is transmitted to the light projecting device 22 from the communication device 14 of the computer device 1 through the optical fiber 22c. The pattern light PL is diffracted by the diffractive optical element 22a and therefore reaches the entire subject to be photographed, which has depth.

[0051] Next, the imaging device 21 of the endoscope 2 captures an image of the object onto which the pattern light PL is projected (step S502). When the object is captured, an imaging signal is transmitted from the imaging device 21 to the communication device 14 of the computing device 1, and the control device 11 generates a captured image DI based on the imaging signal. The generated captured image is stored in the memory 12 or the storage device 13.

[0052] Next, the control device 11 of the computer device 1 identifies the lines (columns) that constitute all first elements of the set of first elements that constitute a pattern in the captured image DI. Similarly, the control device 11 identifies the lines (rows) that constitute all second elements of the set of second elements (step S503). That is, the control device 11 identifies all columns and all rows in the captured image DI. The identified columns and rows are assigned identification numbers (first element identifiers to columns, second element identifiers to rows). As described above, some of the second elements extend intermittently in a two-dimensional plane, but for the intermittently extending elements, linear lines are recognized.

[0053] Next, the control device 11 identifies all intersections of the columns and rows identified in step S503 as nodes (step S504). Next, the control device 11 classifies each node based on its relative position with respect to adjacent nodes (step S505). The identified and classified nodes are labeled based on their relative positions (labels are added to nodes in the captured image DI (classification values ​​are assigned)).

[0054] The processes of steps S503 to S505 are executed based on training data learned through deep learning. In this embodiment, the control device 11 implements U-Net for training. U-Net is a fully convolution network (FCN) that uses deep learning and estimates segmentation within an image. Segmentation refers to the process of dividing an image into multiple regions. U-Net classifies what each pixel represents based on the training data through convolution operations, activation operations (ReLU), max pooling operations, etc.

[0055] FIG. 6 visually illustrates an example of the process of training a U-Net based on a captured image DI. While FIG. 6 presents an example of training a U-Net based on a single image to illustrate an example of training in the U-Net, in practice, the U-Net is trained to identify and classify nodes based on any reference image and any captured image generated by capturing an object onto which patterned light corresponding to the reference image is projected. Note that the example shown in FIG. 6 does not take into account the markers M1 to M9 described above. Also, FIG. 6 shows only some of the patterns in the captured image DI.

[0056] First, as shown in Fig. 6(a), the control device 11 recognizes a captured image DI as learning data. The captured image DI includes a first set of elements SE1 extending vertically in a two-dimensional plane and a second set of elements SE2 extending horizontally in the two-dimensional plane. Note that in the captured image DI, the first set of elements SE1 is distorted due to a deviation in the path of the pattern light caused by the depth of the subject.

[0057] Next, the user adds vertical marks VM to each of the sets of first elements SE1 to label each first element for identification. As shown in FIG. 6(b), the vertical marks VM are added by drawing a curve that traces each first element. The vertical marks VM are recognized as learning data by the control device 11.

[0058] Next, the control device 11 recognizes, for example, a predetermined area of ​​the vertical mark VM and adds a mark. In this embodiment, as shown in Figure 6(c), an area covering a predetermined number of pixels on the right side of the vertical mark VM in the two-dimensional plane is recognized as a vertical mark area VR1, and an area covering a predetermined number of pixels on the left side of the vertical mark VM in the two-dimensional plane is recognized as a vertical mark area VR2. The vertical mark area VR1 and the vertical mark area VR2 are recognized by the control device 11 as learning data.

[0059] Similarly, to add a label for identifying each second element, the user adds a horizontal mark to each of the sets SE2 of second elements. The horizontal mark is added by drawing a curve that traces each second element. The horizontal mark is recognized as learning data by the control device 11.

[0060] Next, the control device 11 recognizes, for example, a predetermined area of ​​the horizontal mark and adds a mark. In this embodiment, as shown in FIG. 6(d), an area covering a predetermined number of pixels above the horizontal mark in the two-dimensional plane is recognized as a horizontal mark area HR1, and an area covering a predetermined number of pixels below the horizontal mark in the two-dimensional plane is recognized as a horizontal mark area HR2. The horizontal mark areas HR1 and HR2 are recognized as learning data by the control device 11.

[0061] As described above, the second element set SE2 includes elements that extend intermittently in a two-dimensional plane. The intermittently extending elements intersect with the same first element at two different points. In the example of FIG. 6(a), intersection point I1 is located lower in the two-dimensional plane than intersection point I2, and intersection point I3 is located lower in the two-dimensional plane than intersection point I4. Horizontal mark areas HR1 and HR2 are drawn as linear lines that cover these intersection points. As shown by the area within the dashed line frame in FIG. 6(d), horizontal mark areas HR1 and HR2 extend upward and to the right in the two-dimensional plane corresponding to intersection points I1 and I2, and extend downward and to the right in the two-dimensional plane corresponding to intersection points I3 and I4.

[0062] Next, the control device 11 assigns an identification number (first element identifier) ​​to the boundary between the vertical mark area VR1 and the vertical mark area VR2 as a first element. The first element to which the first element identifier is assigned is recognized as learning data. Similarly, the control device 11 assigns an identification number (second element identifier) ​​to the boundary between the horizontal mark area HR1 and the horizontal mark area HR2 as a second element. The second element to which the second element identifier is assigned is recognized as learning data. Note that the first element identifier may have any symbol, shape, color, etc. to identify each first element. The same applies to the second element identifier.

[0063] In addition, U-Net may be trained to recognize rows and columns by, for example, displaying a reference image on the output device 16 and having the user input vertical and horizontal marks on the displayed reference image via the input device 15.

[0064] Next, the control device 11 recognizes the intersection of the first element and the second element as a node. Then, the control device 11 classifies all nodes based on their relative positions with respect to adjacent nodes and adds a label (assigns a classification value). In this embodiment, based on the relative positions of two adjacent nodes, the two nodes are classified into three classes: the two nodes are at the same or approximately the same height in a two-dimensional plane, the node located on the left of the two nodes is at a higher position in the two-dimensional plane, or the node located on the right of the two nodes is at a higher position in the two-dimensional plane.

[0065] In practice, it may be impossible to recognize the relative position of a node relative to its adjacent nodes, so such a node may be classified as an unknown class. In this case, the nodes are classified into four classes. In this embodiment, the nodes are classified into four classes. Figure 6(e) shows the labeled nodes.

[0066] Two nodes being at the same or approximately the same height in a two-dimensional plane means, for example, that the difference in height between the two nodes in the two-dimensional plane is within a predetermined threshold range (for example, m pixels (m is an arbitrary number)). Nodes classified in this way are labeled red (R), and in Figure 6(e), a white circle represents the R label.

[0067] The nodes labeled with R correspond to the intersections of the second and first elements that extend continuously in the pattern. In the example shown in Figure 6(a), the second elements that make up the intersections I7 and I8 extend continuously, so nodes N4 and N5 are identified and labeled with R corresponding to the intersections I7 and I8. Similarly, nodes N6 to N11, N14, and N16 to N20 are labeled with R corresponding to the intersections of the second and first elements that extend continuously.

[0068] The fact that the right-side node of two nodes is higher in a two-dimensional plane means, for example, that the right-side node is higher than the left-side node in a two-dimensional plane, and the difference in height between the two nodes in the two-dimensional plane exceeds a predetermined threshold. Nodes classified in this way are labeled green (G), and in Figure 6(e), a black circle represents the G label.

[0069] The nodes to which the G label is added correspond to two intersections between the second element and the first element that extend intermittently in the pattern (the intersection with the second element located on the right is higher than the intersection with the second element located on the left). In the example shown in FIG. 6(a), the second elements that constitute intersections I1 and I2 extend intermittently, and intersection I2 with the second element located on the right is higher than intersection I1 with the second element located on the left. Therefore, a node N1 is identified corresponding to intersections I1 and I2, and a G label is added. Similarly, the second elements that constitute intersections I5 and I6 extend intermittently, and intersection I6 with the second element located on the right is higher than intersection I5 with the second element located on the left. Therefore, a node N3 is identified corresponding to intersections I5 and I6, and a G label is added. Similarly, a G label is added to node N13, corresponding to the intersection of the second element and the first element that extend intermittently.

[0070] The fact that the left node of two nodes is higher in a two-dimensional plane means, for example, that the node on the left is higher in a two-dimensional plane than the node on the right, and the difference in height between the two nodes in the two-dimensional plane exceeds a predetermined threshold. Nodes classified in this way are labeled blue (B), and in Figure 6(e), the shaded circle represents the B label.

[0071] The nodes to which the B label is added correspond to two intersections of the second element and the first element that extend intermittently in the pattern (the intersection with the second element located on the left is located higher than the intersection with the second element located on the right). In the example shown in FIG. 6(a), the second elements that constitute the intersections I3 and I4 extend intermittently, and the intersection I3 with the second element located on the left is located higher than the intersection I4 with the second element located on the right. Therefore, the node N2 is identified corresponding to the intersections I3 and I4, and the B label is added. Similarly, the nodes N12 and N15 correspond to the intersections of the second element and the first element that extend intermittently, and the B label is added.

[0072] In this embodiment, an example has been shown in which nodes are classified into three classes based on their relative positions, but the number of classes to be classified is not limited to three. For example, if both the first element and the second element in the reference image RI include elements that extend intermittently, the nodes may be classified into classes in which the upper node among the adjacent nodes is located to the right (or left) of the lower node in a two-dimensional plane. Note that representing the labels with the above-mentioned colors is merely an example, and any symbol indicating the relative position may be added as a label.

[0073] In this way, the captured image DI is input to U-Net, columns (first elements) and rows (second elements) are identified from the captured image DI, and nodes are identified and classified. Following the above-mentioned procedure, U-Net is trained to recognize columns and rows and identify and classify nodes based on any reference image and a captured image generated by projecting a pattern light corresponding to the reference image onto a target.

[0074] U-Net outputs a first element identifier and a second element identifier (a node is identified by a pair of the first element identifier and the second element identifier) ​​and a label for each node. The first element identifier and the second element identifier for identifying the node are represented as a two-dimensional feature vector. The labels of the three classes (RGB) and the unknown class are represented as a four-dimensional feature vector F.

[0075] As described above, once the nodes have been identified and classified, the control device 11 assigns edges connecting the nodes based on the classification of the nodes (i.e., the positional relationship of the nodes) and the adjacency relationship with adjacent nodes. In the processing of steps S503 to S505, a feature vector F including a six-dimensional (two-dimensional (first element identifier and second element identifier) ​​+ four-dimensional (three classes + unknown class)) vector is extracted from the captured image DI. In addition, a lattice graph G in which labels are added to the nodes is generated.

[0076] Optionally, in the processing of steps S503 to S505, in order to improve the accuracy of the matching, the nodes may be classified based on a predetermined area in which the markers M1 to M9 placed in the reference image RI (the markers M1 to M9 are also placed in the captured image DI) are located, and a label may be added to the node (a classification value may be assigned).

[0077] 7 shows regions R1 to R9, which are predetermined regions centered on markers M1 to M9, respectively, within the reference image RI shown in FIG. 3. For example, region R1 includes four grids formed by a first element and a second element constituting the node where marker M1 is located, and the first elements and second elements adjacent to those elements. Regions R2 to R9 also include four grids. In other words, regions R1 to R9 each include the node where markers M1 to M9 are located, respectively, and the nodes adjacent to those nodes.

[0078] The regions R1 to R9 each have a relationship such as identical, different, symmetrical, and vertically symmetrical in the shape of a lattice formed by the first element and the second element. That is, the relationships between the center node and adjacent nodes in the regions R1 to R9 are identical, different, symmetrical, and vertically symmetrical. Therefore, the nodes in the regions R1 to R9 are classified based on the relationship between the center node and adjacent nodes.

[0079] As shown in FIG. 7, regions R1 to R5 have mutually different shapes. Region R6 has a shape that is vertically and horizontally symmetrical to region R4 in a two-dimensional plane. Region R7 has a shape that is horizontally symmetrical to region R2 in a two-dimensional plane. Region R8 has a shape that is horizontally symmetrical to region R3 in a two-dimensional plane. Region R9 has a shape that is vertically and horizontally symmetrical to region R1 in a two-dimensional plane. These classifications are also recognized as training data by U-Net.

[0080] In the processing of steps S503 to S505, the captured image DI is input to U-Net, and the markers M1 to M9 in the captured image DI are classified into five classes based on the learning data generated by the above-mentioned learning. In the example of Fig. 7, for example, regions R1 and R9 are classified into class 1, regions R2 and R7 into class 2, regions R3 and R8 into class 3, regions R4 and R6 into class 4, and region 5 into class 5.

[0081] In practice, it may be impossible to identify the area surrounding a marker or its shape, and such a marker may be classified as an unknown class. In this case, the markers are classified into six classes. In this embodiment, the markers are classified into six classes.

[0082] These classes can identify the relationship between the node where each marker is located and the nodes adjacent to that node. For example, in region R1, the node below the node where marker M1 is located (the node consisting of intersections I1 and I2 (not shown in Figure 7)) is labeled G (the node located to the right is classified as higher in the two-dimensional plane than the node located to the left). Therefore, by learning these positional relationships for each classified class, it becomes easier to analyze the correspondence between the reference image and an image generated by photographing an object on which a patterned light corresponding to the patterned light to be projected onto the object is projected.

[0083] Through the above-described optional processing, in the processing of steps S503 to S505, a feature vector F including a 12-dimensional (2-dimensional+4-dimensional+6-dimensional (5 classes+unknown class)) vector is extracted from the captured image DI.

[0084] In this embodiment, the first element identifier and the second element identifier are assigned in U-Net, but the method is not limited to this. A number identifying each first element and a number identifying each second element may be assigned in advance, and these numbers (arbitrary symbols identifying the elements) may be embedded in the reference image RI.

[0085] In this embodiment, a set of information indicating corresponding nodes and a set of information indicating corresponding relative positions are extracted from a U-Net that has learned the correspondence between each node in an image and its corresponding relative position, but the neural network used is not limited to a U-Net. For example, a known neural network other than a U-Net that has learned the correspondence between each node in an image and its corresponding relative position may be used.

[0086] Alternatively, two adjacent nodes in an image may be recognized and their relative positions determined (e.g., by image processing such as Hough transform, projective transformation, and / or binarization) without using a neural network. That is, all of the processes in steps S503 to S505 identify respective columns and rows in an image and classify nodes, regardless of whether a neural network is used.

[0087] Returning to the explanation of FIG. 7, the control device 11 associates each node in the grid graph G with a node in the grid graph generated from the corresponding reference image RI based on the feature vector F and the grid graph G (step S506).

[0088] The above-mentioned U-Net generates a graph G based on the results of learning a reference image RI, etc. However, U-Net cannot always generate an accurate graph, and an incorrect graph cannot accurately perform the above-mentioned correspondence.

[0089] In this embodiment, the control device 11 implements a graph convolutional network (GCN) for learning, and the processing of step S506 is executed based on learning data learned in deep learning using the GCN. The GCN is a neural network that applies deep learning to graph data and performs a convolution operation on the graph data. In the convolution operation in the GCN, the likelihood is calculated for each node according to a weight value for each relationship with an adjacent node. The feature vector F described above represents the properties (relative position) of each node itself in the captured image DI, and the lattice graph G represents the adjacent relationships between nodes.

[0090] The GCN is trained based on the feature vectors F and lattice graph G extracted and generated from all reference images and all captured images. Thus, based on the lattice graph G generated from the reference image RI, the GCN is trained to extract nodes corresponding to nodes in the reference image RI. Thus, the output from the GCN is information representing the nodes in the input lattice graph G.

[0091] 8 shows the configuration of a GCN 800 implemented in this embodiment. The GCN 800 performs a predetermined number of calculations on the input feature vector F and lattice graph G, and therefore employs a hierarchical structure according to the number of calculations. The GCN 800 includes a GCN layer 801, a fully connected layer 802, and an output layer 803.

[0092] In the GCN layer 801, a graph convolution calculation unit 801a performs a graph convolution calculation based on the feature vector F and the lattice graph G, a normalization calculation unit 801b performs a normalization calculation (Batch Normalization), and an activation calculation unit 801c performs an activation calculation (ReLU). In this process, the above calculations are performed for each node on the 12-dimensional feature vector F. Thus, a matrix H of D×N (N is the number of nodes, and D is the number of dimensions (12)) is generated.

[0093] The GCN performs calculations for each node based on the relationship with adjacent nodes. In this embodiment, each node in the grid graph G has adjacent nodes in four directions (upward, downward, rightward, and leftward) (it has an adjacent relationship with adjacent nodes in four directions (hereinafter, the number of directions is represented by l)). Therefore, for each direction, a matrix H (1) ~H (l) These matrices are accumulated into the layer data matrix, so that the final matrix H (l+1) is generated. Matrix H (l+1) can be expressed by equation (1).

[0094]

number

[0095]

number

[0096] is the adjacency matrix of the grid graph G with self-joins added, and I is the identity matrix.

[0097]

number

[0098] teeth,

[0099]

number

[0100] is the order matrix of W (l) is the weight matrix of this layer, and σ is the activation function (ReLU). The adjacency matrix A is {A0 (upward), A1 (downward), A2 (rightward), A3 (leftward)}.

[0101] matrix H (l+1) The calculation of is performed according to equation (2).

[0102]

number

[0103]

number

[0104] is a weight matrix according to the direction d∈{0, 1, 2, 3}.

[0105] After executing equation (2), normalization and activation operations are performed. This process is repeated five times. In this way, the GCN layer 801 generates the matrix H (l+1) Output.

[0106] In the fully connected layer 802, the matrix H (l+1) Based on this, the linear transformation calculation unit 802a performs a linear transformation calculation, and the activation calculation unit 802b performs an activation calculation (ReLU). In this process, the above calculation is performed for each of D dimensions (12 dimensions in this embodiment) and N nodes. Thus, a feature embedding matrix (having a size of N×D) for the feature vector F is generated. In this way, the fully connected layer 802 outputs the feature embedding matrix for the feature vector F.

[0107] The above-mentioned calculations allow for a convolution operation that takes into account each node's own feature vector (i.e., a value indicating its relative position with respect to adjacent nodes), its relationship with adjacent nodes, and weighting, thereby improving the accuracy of the correspondence with each node in the original reference image RI. Figure 9 visually illustrates the calculation that takes into account the node's feature vector, its relationship with adjacent nodes, and weighting.

[0108] In the output layer 803, a Softmax calculation unit 803a performs Softmax calculation for each node on the feature embedding matrix, and outputs a log-likelihood vector for each node.

[0109] The GCN is trained based on the lattice graph G output from the U-Net. Specifically, the nodes in the lattice graph G that are identified and labeled by the U-Net are recognized as training data. A first element identifier is assigned to identify each first element in the lattice graph, and a second element identifier is assigned to identify each second element, and the first element identifier and the second element identifier are used as training data.

[0110] When the GCN outputs a log-likelihood vector for each node, the corresponding node is determined based on the log-likelihood vector. In this determination, a node determined to have a certain probability from the training data is extracted as a node corresponding to a node in the reference image RI. The GCN outputs a set of first element identifiers and a set of second element identifiers that reflect the nodes after the above-mentioned calculation. The first set of element identifiers and the second set of element identifiers make it possible to recognize columns and rows in the image, and therefore identify nodes that are their intersections.

[0111] In this embodiment, the GCN outputs a set of first element identifiers and a set of second element identifiers in consideration of the load of outputting all nodes, but it is also possible to assign identifiers to all nodes (node ​​identifiers) and output the node identifiers.

[0112] Although nodes corresponding to nodes in the reference image are extracted from a GCN trained based on feature vectors for nodes in the pattern and a lattice graph formed by the nodes, the neural network used is not limited to a GCN. For example, a known neural network other than a GCN trained based on a lattice graph formed by feature vectors and nodes may be used. A function for aggregating feature vectors associated with nodes in a graph based on graph connections may also be used, and other methods and neural networks may be used instead. Even when a network other than a GCN is used, calculations are performed for each node in the lattice graph taking into account the relationship with adjacent nodes and weight values.

[0113] Alternatively, without using a neural network, nodes corresponding to nodes in the reference image may be extracted from a lattice graph formed by nodes based on the relative positions of the nodes in the captured image to adjacent nodes (classification based on relative positions) and the relationships with the adjacent nodes (for example, by image processing such as spatial filtering and convolution operations). In other words, regardless of whether a neural network is used, the processing in step S506 extracts nodes corresponding to nodes in the reference image based on the input lattice graph and feature vectors (classification based on the relative positions of the nodes in the lattice graph to adjacent nodes) while taking into account the relationships with adjacent nodes and weight values.

[0114] In this way, the GCN is used to identify each node in the captured image DI that corresponds to each node in the reference image RI. Since more nodes in the captured image DI are associated with nodes in the reference image RI in this way, the distance between the imaging device and the subject can be calculated based on the associations using triangulation, and a three-dimensional image can be constructed based on this distance. Furthermore, external parameters (used in epipolar geometry) indicating positional information between the imaging device 21 and the light projector 22 can be calculated based on the associations.

[0115] As described above, the shape of the reflected pattern light is distorted depending on the depth of the object being photographed, making it difficult to associate all nodes in an image with the original nodes using image processing techniques of the prior art. In this embodiment, even if the shape of the pattern is distorted due to elements extending intermittently vertically and / or horizontally in a two-dimensional plane, the relationship between adjacent nodes in the captured image DI is maintained. Based on this relationship, the accuracy of associating nodes in the original pattern using a GCN can be improved. For example, in the captured image DI photographed by projecting the reference image RI shown in FIG. 3, even if the shape of the pattern is distorted, the height relationship in the horizontal direction in the two-dimensional plane is maintained between two nodes.

[0116] In addition, since the reference image RI is configured by including elements that extend intermittently in at least one of the first element and the second element, the information required for matching nodes can be reduced. Furthermore, since the adjacency relationship of nodes is determined based on classification based on marks M1 to M9 placed in the reference image RI, the accuracy of matching can be further improved.

[0117] In addition to the above-described processing, the relative positions of pixels in the captured image DI and adjacent nodes may be determined, and the relative positions of pixels in the reference image RI and adjacent nodes may be determined, and pixel-by-pixel correspondence may be performed based on the relative positions of both. This processing recognizes the relative positions of each pixel with adjacent nodes for both the captured image DI and the reference image RI, which increases the processing load, but can further improve the accuracy of correspondence. Such pixel-by-pixel correspondence is also learned by the GCN.

[0118] The correspondence between nodes based on the output from the GCN and the relative position of the node to its neighbors described above may be performed by a GCN or other neural network, or by image processing without a neural network (e.g., spatial filtering and convolution operations).

[0119] <Second embodiment> Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that a reference image RI is also input to the U-Net and GCN together with the captured image DI, and the outputs from both GCNs are compared.

[0120] An example of processing executed by the image analysis system 100 according to the second embodiment will be described with reference to Fig. 10. Steps S1001 to S1005 shown in Fig. 10 are the same as steps S501 to S505 shown in Fig. 5, and therefore a description thereof will be omitted. Note that from step S1005 onwards, a feature vector F d and the grid graph G d will be output.

[0121] In step S1006, the control device 11 inputs the reference image RI stored in the storage device 13 to the U-Net, and identifies the lines (columns) that make up the first element and the lines (rows) that make up the second element in the reference image RI. The method for identifying columns and rows is the same as the method described for step S503 shown in Figure 5.

[0122] Next, the control device 11 identifies all the intersections of the columns and rows identified in step S1006 as nodes (step S1007). The method for identifying the nodes is the same as the method described for step S504 shown in FIG.

[0123] Next, the control device 11 classifies each node based on its relative position with respect to its adjacent nodes (step S1008). The method for classifying the nodes is the same as the method described for step S505 shown in FIG. 5. From step S1008, the feature vector F p and the grid graph G p will be output.

[0124] Next, the control device 11 receives the output of step S1005 (feature vector F d and the grid graph G d ) is input to the GCN, and the feature embedding matrix F calculated from the captured image DI is d (Step S1009). The feature embedding matrix F d The method for calculating is the same as that described for step S506 shown in FIG.

[0125] Next, the control device 11 receives the output of step S1008 (feature vector F p and the grid graph G p ) is input to the GCN, and the feature embedding matrix F calculated from the reference image RI is used. d (Step S1010). The feature embedding matrix F p The method for calculating is the same as that described for step S506 shown in FIG.

[0126] Next, the control device 11 calculates the feature embedding matrix F d and the feature embedding matrix F p For each node, the feature vector F d and F d The similarity between the reference image RI and the captured image DI is determined by calculating the inner product of the feature vector F d and F d The dot product of

[0127]

number

[0128] is expressed by

[0129] The control device 11 uses a Softmax function to derive the Softmax value of the inner product. p and G d Based on the above, the above-described calculation is performed to learn by using a cost function of the dot product of the two feature vectors and the cross entropy between the first element identifier and the second element identifier. From the above-described learning, a Softmax value is derived for each node using the Softmax function of the dot product, and a log-likelihood vector for each node is evaluated, thereby making it possible to find correspondence between two images for each node.

[0130] The process of the second embodiment is basically the same as that of the first embodiment, but it also performs calculations on the reference image via the GCN, calculates the inner product of the feature vectors of both the reference image RI and the captured image DI, and evaluates the log-likelihood vector for each node. Figure 11 visually represents this process.

[0131] As described above, the second embodiment has been described. The second embodiment also makes it possible to improve the accuracy of associating nodes in a captured image with nodes in a reference image.

[0132] <Third embodiment> Next, a third embodiment will be described. The third embodiment differs from the first and second embodiments in the reference image RI used.

[0133] A reference image RI according to the third embodiment will be described with reference to Fig. 12. As shown in Fig. 12, the reference image RI includes a set of rectangular elements extending continuously in the vertical direction on a two-dimensional plane and a set of arbitrary symbol elements (in this embodiment, crosses) extending intermittently in the vertical direction on the two-dimensional plane.

[0134] As shown in Fig. 12, first elements (shaded rectangles shown in Fig. 12) E1a to E1n (n is any integer) in the first element set SE1 extend continuously in the vertical direction in a two-dimensional plane in the reference image RI. Meanwhile, second elements (cross symbols shown in Fig. 12) E2a to E2n (n is any integer) in the second element set SE2 extend intermittently in the vertical direction in the two-dimensional plane in the reference image RI. The second elements E2a to E2n are respectively arranged along the first elements E1a to E1n. Fig. 12 shows an enlarged view of a portion of the reference image RI surrounded by the circle C1 within a circle C2 corresponding to the circle C1 enclosing the portion of the reference image RI.

[0135] In the area within the circle C2, the first elements E1a to E1c each extend continuously in the vertical direction. The second elements E2a to E2n are arranged at random intervals in the vertical direction. For example, the distance between the second elements E2b and E2e is different from the distance between the second elements E2e and E2h. Similarly, the distance between the second elements E2c and E2f is different from the distance between the second elements E2f and E2i.

[0136] Because the second elements E2a to E2n are arranged at random intervals in the vertical direction in the two-dimensional plane, the angle at which a first second element is positioned relative to an adjacent second element in the two-dimensional plane is different from the angle at which a second second element is positioned relative to an adjacent second element in the two-dimensional plane. For example, in the area within circle C2, the angle at which second element E2b is positioned relative to the adjacent second element E2c is different from the angle at which second element E2c is positioned relative to the adjacent second element E2d. Similarly, the angle at which second element E2b is positioned relative to the adjacent second element E2c is different from the angle at which second element E2h is positioned relative to the adjacent second element E2i.

[0137] Although the present embodiment employs a reference image including a set of rectangular elements extending continuously in the vertical direction in a two-dimensional plane and a set of symbol elements extending intermittently in the vertical direction in the two-dimensional plane, the reference image is not limited to such a configuration. For example, although not shown, the reference image may include a set of rectangular elements extending continuously in the horizontal direction in a two-dimensional plane and a set of symbol elements extending intermittently in the horizontal direction in the two-dimensional plane. In this case, the set of symbol elements is arranged at random intervals in the horizontal direction.

[0138] Although not shown, n markers may be arranged in the reference image RI shown in FIG. 12 as well, similarly to the reference image RI shown in FIG.

[0139] For the reference image RI used in the third embodiment, a corresponding pattern light PL is projected onto the object to generate a captured image DI. The method for processing the captured image DI according to the third embodiment is similar to either of the methods described in the first and second embodiments, but differs from the processing according to the first and second embodiments in that it identifies only columns in the image and classifies nodes based on the relative angles between adjacent nodes.

[0140] As described above, in the reference image RI used in the third embodiment, both the first element and the second element extend vertically in a two-dimensional plane, so only columns in the image are identified in the captured image DI. Columns are identified, for example, by tracing the edge of the first element or the second element (symbol) and drawing a linear line that extends vertically in the two-dimensional plane. This processing is similar to the method described with reference to FIGS. 6(b) and 6(c).

[0141] Furthermore, regarding the identification of a node, for example, as shown in the rectangular area labeled NR1 in Fig. 12, a certain area between a first element and a second element is identified as a node. That is, the area is divided into certain areas based on the classified labels (relative angles (feature vectors) to adjacent nodes) described later. The node area NR1 in Fig. 12 is divided as a certain area adjacent to the second element E2f on the first element E1b, and the area NR1 is identified as a node.

[0142] Similarly, the node region NR2 is divided into a fixed region on the first element E1b adjacent to the second element E2i, and the region NR2 is identified as a node. The node region NR3 is divided into a fixed region on the first element E1c adjacent to the second element E2g, and the region NR3 is identified as a node. The divided regions form multiple lattices, each containing a node. Such node regions may also be identified by the above-mentioned U-Net learning. The node regions shown in FIG. 12 are merely examples, and a fixed region between the first element and the second element may be identified as a node according to a predetermined rule.

[0143] In the captured image DI generated from the subject onto which the pattern light PL corresponding to the reference image RI used in the third embodiment is projected, a certain region between the first element and the second element is also identified as a node as described above. In the third embodiment, there is no need to identify rows when identifying nodes, which simplifies the calculation process using U-Net or the like.

[0144] In the third embodiment, nodes are not identified by the intersections of columns and rows, as in the processing in the first embodiment. Measurement accuracy can be improved by projecting the pattern shown in FIG. 12 and capturing an image while calculating the correspondence for each pixel from the captured image. To achieve this, U-Net may be used to learn to extract the grid phase from the captured image, that is, the relative position of each pixel relative to the grid. FIG. 13 shows an example of estimating the grid phase (the relative positional relationship between the repeating grid and each pixel expressed as a rotation angle between 0 and 2π) from the captured image.

[0145] Figure 13 shows an example of grid information for a captured image, where the relative positional relationship between the grid and each pixel is expressed as a rotation angle. This rotation angle is linked to the repetition of the grid, rotating once for each grid. In other words, it increases from 0 to 2π and then returns to 0. Figure 13(a) shows an image projected with the pattern shown in Figure 12. Figure 13(b) shows an image in which, as described above, U-Net estimated a cosine signal such that the rotation angle becomes 0 at the grid division positions. Figures 13(b) and 13(c) show images in which U-Net estimated a cosine signal such that the rotation angle becomes 4π / 5 at the grid division positions from the captured image shown in Figure 13(a). Figure 13(d) shows the results of calculating grid phase information for each pixel from the phase estimation results including Figures 13(b) and 13(c). Figure 13(e) shows the results of calculating the vertical phase information of the grid for each pixel.

[0146] The estimation of phase information (rotation angle) of the captured image shown in FIG. 13 is merely an example, and it is also possible to directly detect the phase information of the lattice using U-Net without estimating the cosine signal. However, learning to detect the cosine signal will result in higher accuracy in estimating the phase information than directly detecting the phase information using U-Net. Gabor filters, etc., can also be used. As mentioned above, the phase information of the lattice may be identified using a neural network other than U-Net or other image processing.

[0147] In this embodiment, the nodes of the graph are identified by dividing the phase information for each pixel shown in FIG. 13 into regions using 0-degree phase portions as region separators (i.e., dividing the captured image into multiple regions based on the relative positions of each pixel and grid information on the captured image). The adjacency relationships between the nodes are also extracted from the adjacency relationships between the regions. Once the nodes are identified, the control device 11 classifies each node based on image features around the node using a U-Net or the like, and assigns a label (assigns a classification value). Once the nodes are identified and classified, the control device 11 assigns edges connecting the nodes based on the adjacency relationships with adjacent nodes.

[0148] Note that when extracting nodes through region division, information other than topological information may be used. For example, a certain range from the lattice may be recognized as a node using U-Net, and region division may be performed using techniques such as Delaunay division with that node at the center. Furthermore, region division based on topological information may be combined with region division centered on the node. Furthermore, connections between nodes may be based on proximity relationships in addition to adjacency relationships. In this case, the output graph will be a general graph rather than a lattice graph. Figure 14 shows an example of connections based on proximity relationships.

[0149] In this embodiment, for each node, based on the relative positions of the two points associated with that node, the two points are classified into five classes: the two points are horizontal or approximately horizontal in a two-dimensional plane; the point located on the left of the two points is at a positive angle in the two-dimensional plane relative to the point located on the right, and that angle exceeds a predetermined angle; the point located on the left of the two points is at a positive angle in the two-dimensional plane relative to the point located on the right, and that angle is equal to or smaller than a predetermined angle; the point located on the left of the two points is at a negative angle in the two-dimensional plane relative to the point located on the right, and that angle is equal to or smaller than a predetermined angle; and the point located on the left of the two points is at a negative angle in the two-dimensional plane relative to the node located on the right, and that angle exceeds a predetermined angle.

[0150] A positive angle is an angle where the left of two adjacent points is lower on a two-dimensional plane than the right of the point, and a negative angle is an angle where the left of two adjacent points is higher on a two-dimensional plane than the right of the point.

[0151] In practice, it may be impossible to recognize the relative positions of two points associated with a certain node, so such a node may be classified as an unknown class. In this case, the nodes are classified into six classes. In this embodiment, the nodes are classified into six classes. Figure 15 shows labeled nodes.

[0152] The fact that the left point of two points is at a positive angle relative to the right point on a two-dimensional plane and that the angle exceeds a predetermined angle is equivalent to, for example, the fact that the left point is lower in the two-dimensional plane than the right point and the difference in height between the two points on the two-dimensional plane exceeds a predetermined threshold. Points classified in this way are labeled green (G), and in FIG. 15, a black circle represents the G label. In the example shown in FIG. 15, point N1 is lower in the two-dimensional plane than point N6, and the G label is assigned assuming that the angle of point N1 relative to point N6 exceeds a threshold.

[0153] The fact that the left point of two points is at a positive angle relative to the right point on a two-dimensional plane and that the angle is equal to or less than a predetermined angle means, for example, that the left point is lower in the two-dimensional plane than the right point and the difference in height between the two points on the two-dimensional plane is equal to or less than a predetermined threshold. Points classified in this way are labeled blue (B), and in FIG. 15, a shaded circle represents the B label. In the example shown in FIG. 15, point N2 is lower in the two-dimensional plane than point N7, and the B label is assigned assuming that the angle of point N2 relative to point N7 is equal to or less than a threshold.

[0154] Two points being horizontal or nearly horizontal on a two-dimensional plane means, for example, that the difference in height between the two points on the two-dimensional plane is within a predetermined threshold range. Points classified in this way are labeled yellow (Y), and in Figure 15, a shaded circle (brighter than the B label) represents the Y label. In the example shown in Figure 15, point N3 is located horizontally or nearly horizontally with point N8, so the Y label is added.

[0155] The fact that the left point of two points is at a negative angle relative to the right point on a two-dimensional plane and that the angle is equal to or less than a predetermined angle means, for example, that the left point is higher on the two-dimensional plane than the right point and the difference in height between the two points on the two-dimensional plane is equal to or less than a predetermined threshold. Points classified in this way are labeled purple (P), and in FIG. 15, a shaded circle (lighter than the Y label) represents the P label. In the example shown in FIG. 15, point N4 is higher on the two-dimensional plane than point N9, and the P label is assigned assuming that the angle of point N4 relative to point N9 is equal to or less than a threshold.

[0156] The fact that the left point of two points is at a negative angle relative to the right point on a two-dimensional plane and that the angle exceeds a predetermined angle is equivalent to, for example, the fact that the left point is higher on the two-dimensional plane than the right point and the difference in height between the two points on the two-dimensional plane exceeds a predetermined threshold. Points classified in this way are labeled red (R), and in FIG. 15, a white circle represents the R label. In the example shown in FIG. 15, point N5 is higher on the two-dimensional plane than point N10, and the R label is assigned assuming that the angle of point N5 relative to point N10 exceeds a threshold.

[0157] In the third embodiment, after determining the correspondence for each node as in the first embodiment, the correspondence information for each node is combined with the phase information determined by U-Net to determine the correspondence information for each pixel. Specifically, by setting the correspondence information for each node adjacent to a pixel as an integer value and the phase information as a decimal value between 0 and 1, and adding them together, the correspondence information for each pixel can be determined with real number precision.

[0158] In this embodiment, an example has been shown in which nodes are classified into five classes based on the relative angles between horizontally adjacent points, but the number of classes to be classified is not limited to five. For example, if both the first element and the second element in the reference image RI include elements that extend intermittently, they may be further classified into classes based on the relative angles between vertically and horizontally adjacent points. Note that representing the labels with the above-mentioned colors is merely an example, and any symbol indicating the relative position may be added as a label.

[0159] For the reference image RI shown in the third embodiment, even if the corresponding pattern light PL is projected onto the object to be photographed and the shape of the pattern is distorted, the relationship with the adjacent nodes is maintained in the photographed image DI. Therefore, based on the relationship, it is possible to improve the accuracy of associating the nodes with the nodes in the original pattern using GCN.

[0160] In the third embodiment, correspondence between nodes may also be performed based on the output from the GCN and the relative angles between points associated with the nodes (classification values ​​assigned based on the relative angles) (by using a GCN or other neural network, or by image processing that does not use a neural network (e.g., spatial filtering and convolution operations, etc.)).

[0161] Note that instead of the reference image RI shown in Figures 3, 4, and 12, the pattern may be configured with a pattern including a square grid. Although such a pattern itself cannot represent the above-mentioned differences in relative position and relative angle, the above-mentioned differences may be represented by placing any symbols representing the relative position and relative angle within the grid. In other words, the reference image RI represents the above-mentioned differences in relative position and / or relative angle in either the first element or the second element.

[0162] It should be noted that the hardware components described in the above embodiments are merely exemplary, and other configurations are possible. Furthermore, the order of the processes described in the above embodiments does not necessarily have to be the order described, and may be performed in any order. Furthermore, additional steps may be newly added without departing from the basic concept of the present invention.

[0163] Furthermore, the image analysis method according to one embodiment of the present invention is implemented by a computer program executed by the control device 11 (processor) of the computer device 1, and the computer program may be stored in a non-transitory storage medium. Examples of non-transitory storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disk devices, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims

1. A computer-implemented method for analyzing a correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically in a two-dimensional plane and a second set of elements extending horizontally in the two-dimensional plane, and the reference image represents that at least one of the first set of elements and the second set of elements includes elements extending intermittently; identifying, within the captured image, respective nodes formed from the first set of elements and the second set of elements; assigning a classification value to each of the nodes by classifying the node based on its relative position to adjacent nodes; generating a graph from the classified nodes; For each node in the graph, determining its adjacency with adjacent nodes; matching nodes in the graph with nodes in the reference image based on the classification values ​​and the adjacency relationships; A method comprising: The reference image includes n markers, and the method comprises: identifying n regions each containing said n markers; For each of the n regions, classifying the nodes based on a relationship between a central node and adjacent nodes within the region and assigning a second classification value; matching nodes in the graph with nodes in the reference image further based on the second classification value; The method further comprising:

2. the step of associating nodes in the graph with nodes in the reference image is performed by using a neural network, the neural network being trained to output correspondences of nodes in the graph to nodes in the reference image based on the classification values ​​and the adjacency relationships.

2. The method of claim 1.

3. The neural network is a graph convolutional network (GCN).

3. The method of claim 2.

4. If the second set of elements represents intermittently extending elements, the reference image is a plurality of second elements in the set of second elements intersect with a single first element in the set of first elements at a plurality of locations; a first height in a two-dimensional plane of an intersection between a first element of the plurality of second elements and the first element is different from a second height in a two-dimensional plane of an intersection between a second element of the plurality of second elements and the first element; 3. The method according to claim 1 or 2.

5. the relative position is based on the first height being different from a second height represented by the reference image; 5. The method of claim 4.

6. In each of the n regions, For a lattice consisting of the central node and adjacent nodes in the region, Based on whether the shapes of the lattices are the same, different, and symmetrical, The second classification value is assigned.

2. The method of claim 1 .

7. the steps of identifying nodes, assigning classification values, and generating a graph are performed by using a second neural network, the second neural network having trained the reference images to identify nodes from the reference images, assign classification values, and generate a graph.

7. The method according to any one of claims 1 to 6.

8. The second neural network is a U-Net.

8. The method of claim 7.

9. For each pixel in the captured image, determining a relative position of the identified nodes with respect to nodes that are proximate to the pixel; Corresponding pixels in the captured image to pixels in the reference image based on the determined relative positions and relative positions of the pixels in the reference image relative to nodes adjacent to the corresponding pixels; Further equipped with 9. The method according to any one of claims 1 to 8, characterized in that

10. identifying respective nodes in the reference image that are comprised of the first set of elements and the second set of elements; assigning a third classification value to each of the nodes by classifying the node based on its relative position to adjacent nodes; generating a second graph from the classified nodes; and for each node in the second graph, determining an adjacency relationship with adjacent nodes; The step of associating nodes in the graph with nodes in the reference image comprises: generating a first feature vector for each node in the graph by inputting the graph into a neural network; generating a second feature vector for each node in the second graph by inputting the second graph into the neural network; calculating, for each node, an inner product value of the first feature vector and the second feature vector; evaluating the dot product value; Contains 10. The method according to any one of claims 1 to 9.

11. A computer-implemented method for analyzing a correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically or horizontally in a two-dimensional plane and a second set of elements extending intermittently in the same direction as the first set of elements, and the reference image represents each of the second set of elements being arranged at random intervals in the same direction; identifying, within the captured image, respective nodes formed from the first set of elements and the second set of elements; assigning a classification value to each of the nodes by classifying the node based on its relative angle to adjacent nodes; generating a graph from the classified nodes; For each node in the graph, determining its adjacency with adjacent nodes; matching nodes in the graph with nodes in the reference image based on the classification values ​​and the adjacency relationships; A method comprising:

12. The step of identifying the node is performed by using a neural network, and the neural network is trained on the captured image and the captured image rotated at a predetermined angle to identify the first element or the second element.

12. The method of claim 11 .

13. the reference image represents that an angle of a first element of the set of second elements relative to an adjacent second element is different from an angle of a second element of the set of second elements relative to an adjacent second element; 13. The method according to claim 11 or 12.

14. The step of generating the graph includes a step of dividing the captured image into a plurality of regions including the nodes based on the relative positions of each pixel in the captured image and grid information on the captured image.

14. The method according to any one of claims 11 to 13.

15. A computer device that analyzes a correspondence relationship between a reference image including a pattern and a captured image generated from a subject onto which pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically in a two-dimensional plane and a second set of elements extending horizontally in the two-dimensional plane, and the reference image indicates that at least one of the first set of elements and the second set of elements includes elements that extend intermittently; Identifying nodes in the captured image that are each made up of the first set of elements and the second set of elements; assigning a classification value to each of the nodes by classifying the node based on its relative position to adjacent nodes; generating a graph from the classified nodes; For each node in the graph, determining an adjacency relationship with adjacent nodes; Corresponding nodes in the graph with nodes in the reference image based on the classification values ​​and the adjacency relationships; a control device configured to The reference image includes n markers, and the control device identifying n regions each containing said n markers; For each of the n regions, classifying the nodes based on a relationship between a central node and adjacent nodes within the region and assigning a second classification value; Corresponding nodes in the graph with nodes in the reference image further based on the second classification value.

1. A computing device comprising:

16. A computer device that analyzes a correspondence relationship between a reference image including a pattern and a captured image generated from a subject onto which pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically or horizontally in a two-dimensional plane and a second set of elements extending intermittently in the same direction as the first set of elements, and the reference image represents each of the second set of elements being arranged at random intervals in the same direction; Identifying nodes in the captured image that are each made up of the first set of elements and the second set of elements; assigning a classification value to each of the nodes by classifying the node based on its relative angle to adjacent nodes; generating a graph from the classified nodes; For each node in the graph, determining an adjacency relationship with adjacent nodes; Corresponding nodes in the graph with nodes in the reference image based on the classification values ​​and the adjacency relationships; a control device configured to The reference image includes n markers, and the control device identifying n regions each containing said n markers; For each of the n regions, classifying the nodes based on a relationship between a central node and adjacent nodes within the region and assigning a second classification value; Corresponding nodes in the graph with nodes in the reference image further based on the second classification value.

1. A computing device comprising:

17. A computer program comprising computer-executable instructions, which when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 14. A computer program characterized by:

Images