Calculation method, projection method, and control device

Abstract

To solve such a problem that, when convolution operations are executed by using the same convolution neural network on both a first image and a second image, the accuracy of the correspondence between the first image and the second image determined by a matching unit becomes low.SOLUTION: A calculation method includes: performing a first convolution operation on a material image MI for projection onto an object, input to a first CNN 124b[1], to extract multiple first feature points CP1; performing a second convolution operation on an object image OI including the object, input to a second CNN 124b[2], to extract multiple second feature points CP2; and performing a third convolution operation on multiple first corresponding points RP1 included in the multiple first feature points CP1 and on multiple second corresponding points RP2 included in the multiple second feature points CP2, input to a third CNN 124b[3], to derive correspondences between the multiple first corresponding points RP1 and the multiple second corresponding points RP2.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a calculation method, a projection method, and a control device. [Background technology]

[0002] Conventionally, techniques for calculating pixel-by-pixel correspondences between multiple images have been used. For example, Patent Document 1 discloses an environment recognition device that includes a feature extraction unit that performs convolution operations on both a first image and a second image using the same convolutional neural network to obtain feature maps for each image, and a matching unit that determines the correspondences between the first image and the second image based on the feature maps. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-180611 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the environment recognition device disclosed in Patent Document 1, a single feature extraction unit performs convolution operations on both the first image and the second image using the same convolutional neural network, resulting in low accuracy in the correspondence between the first image and the second image determined by the matching unit. [Means for solving the problem]

[0005] A calculation method according to one aspect of the present invention includes: extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, which is input to a first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object, which is input to a second CNN; and deriving correspondences between a plurality of first corresponding points and a plurality of second corresponding points by performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, which are input to a third CNN.

[0006] Furthermore, a projection method according to one aspect of the present invention includes: extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, the material image being input to a first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object, the object being input to a second CNN; deriving a correspondence relationship between the plurality of first corresponding points and the plurality of second corresponding points by performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, the third CNN; performing geometric correction on the material image using the correspondence relationship; and projecting a projection image obtained by converting a coordinate system in the geometrically corrected material image into a panel coordinate system using an optical device.

[0007] Furthermore, a control device according to one embodiment of the present invention extracts a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, which is input to a first CNN; extracts a plurality of second feature points by performing a second convolution operation on an object image including the object, which is input to a second CNN; and derives correspondences between the plurality of first corresponding points and the plurality of second corresponding points by performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, which are input to a third CNN. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of a projection system 1. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a control device 10. [Figure 3] 10 is a diagram showing an example of the specifications of information input and output to a matching processing unit 124. FIG. [Figure 4] FIG. 10 is a diagram showing an example of the configuration in which a matching processing unit 124 realizes a feature point extraction function and a matching function using a single CNN. [Figure 5] FIG. 10 is a diagram showing an example of the configuration of the matching processing unit 124 in which CNNs are assigned to each of the feature point extraction function and the matching function. [Figure 6] FIG. 10 is a diagram showing an example of the configuration of the matching processing unit 124, in which a CNN is assigned to each of the feature point extraction function and the matching function, and a mechanism is added to transmit the features of each layer of the CNN that realizes the feature point extraction function to the identification layer. [Figure 7] FIG. 10 is a diagram showing an example of the configuration of a matching processing unit 124, in which a learning model for image classification is used in a CNN that realizes a feature point extraction function. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of the matching processing unit 124 in which a feature pyramid mechanism (FPN) is added to the CNN that realizes the feature point extraction function. [Figure 9] FIG. 10 is a diagram showing an example of the configuration of the matching processing unit 124, in which a matching mechanism (MMNet) that executes matching processing for each resolution is added to the CNN that realizes the matching function. [Figure 10] FIG. 10 is a diagram showing an example of the configuration of a matching processing unit 124 in which a CNN that realizes a feature point extraction function is equipped with an attention mechanism. [Figure 11] FIG. 10 is a diagram showing an example of the configuration of a matching processing unit 124 in a configuration in which a feature point extraction function and a matching function are realized by a single CNN. [Figure 12] FIG. 10 is a diagram showing an example of the configuration of a matching processing unit 124 in a configuration in which a CNN is assigned to each of a feature point extraction function and a matching function. [Figure 13]10 is a graph showing the relationship between the number of layers of CNN in the entire matching processing unit 124 and the average positional deviation. [Figure 14] 10 is a graph showing the relationship between the number of channels in each hidden layer in the matching processing unit 124 and the average positional deviation. [Figure 15] 10 is a graph showing the relationship between the presence and position of a normalization layer and the average positional deviation. [Figure 16] 10 is a graph showing the relationship between kernel size and average positional deviation. [Figure 17] FIG. 10 is a diagram showing an example of the configuration of a matching processing unit 124 in a configuration in which a mechanism for transmitting the features of each layer to a discrimination layer is added to a CNN that realizes a feature point extraction function. [Figure 18] FIG. 10 is a diagram showing an algorithm executed by a matching processing unit 124A according to a first comparative example. [Figure 19] FIG. 10 is a diagram showing an algorithm executed by a matching processing unit 124B according to a second comparative example. [Figure 20] FIG. 2 is a block diagram showing an example of the configuration of a projector 20. [Figure 21] 3 is a flowchart showing the overall processing of the control device 10. [Figure 22] 10 is a flowchart showing the matching process in step S24. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, in each drawing, the dimensions and scale of each part are appropriately different from those of the actual parts. Furthermore, since the embodiments described below are preferred specific examples of the present invention, various technically preferable limitations are applied, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

[0010] 1: First embodiment 1-1: Configuration of the first embodiment The calculation method and projection method according to this embodiment will be described below with reference to FIGS.

[0011] 1-1-1: Overall configuration of the projection system 1 is a block diagram showing the configuration of a projection system 1 that executes a calculation method and a projection method according to this embodiment. The projection system 1 includes a control device 10 and a projector 20. The control device 10 and the projector 20 are connected so that they can communicate with each other.

[0012] The projector 20 is a device that projects a projection image PI onto an object. The object has a three-dimensional shape. Projection mapping is achieved by projecting the projection image PI onto the surface of the object. An object image OI representing the object is, for example, a monochrome three-dimensional image having the shape of a mannequin wearing clothing. The projection image PI is also an image showing the appearance of the clothing.

[0013] The control device 10 is a device that controls the projector 20. More specifically, the control device 10 performs geometric correction on the material image MI showing the above-mentioned clothes, and then performs coordinate transformation to generate the above-mentioned projection image PI. The control device 10 also causes the projector 20 to project the generated projection image PI onto the above-mentioned object. The material image MI may be an image that the control device 10 acquires from a server via a network, or may be an image stored in a storage device 140 described below.

[0014] For this reason, the control device 10 extracts feature points necessary for fitting between the object image OI and the material image MI, which look different from each other. The feature points are, for example, points that indicate the contours of the parts that make up the object and the material, or changes in unevenness. The control device 10 also derives a correspondence relationship between the two extracted feature points. The control device 10 uses the correspondence relationship to perform geometric correction on the material image MI, and converts the coordinate system of the geometrically corrected material image MI into the panel coordinate system of the projector 20, thereby generating the projection image PI. The coordinate system of the material image MI is a coordinate system that indicates the coordinates of the pixels that make up the material image MI, and the panel coordinate system is a coordinate system that indicates the coordinates of the pixels of the liquid crystal panel provided in the projector 20.

[0015] 1-1-2: Control device configuration 1-1-2-1: Overall configuration of the control device 2 is a block diagram showing an example of the configuration of the control device 10. The control device 10 is typically a PC (Personal Computer), but is not limited to this and may be, for example, a tablet terminal or a smartphone. The control device 10 includes an imaging device 110, a processing device 120, a storage device 140, a display device 150, an input device 160, and a communication device 170. The elements of the control device 10 are connected to each other by one or more buses for communicating information.

[0016] The imaging device 110 is a device that captures an image of an object. The imaging device 110 captures images of various objects under the control of the processing device 120. For example, cameras provided in PCs, tablet terminals, and smartphones are preferably used as the imaging device 110, but the imaging device 110 is not limited to these and may also be an external camera such as a web camera.

[0017] The processing device 120 is a processor that controls the entire control device 10, and is configured, for example, by one or more chips. The processing device 120 is configured, for example, by a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 120 may be realized by hardware such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The processing device 120 executes various processes in parallel or sequentially.

[0018] The storage device 140 is an example of a recording medium that can be read and written by the processing device 120, and stores a plurality of programs including the control program PR1 executed by the processing device 120, and a material image database MDB. The storage device 140 may be configured with at least one of, for example, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), and a random access memory (RAM). The storage device 140 may also be called a register, a cache, a main memory, a primary storage device, or the like.

[0019] The material image database MDB is a database that stores the first material image MI1. The first material image MI1 is an image that is the basis of the projection image PI that is projected onto an object.

[0020] The display device 150 is a device that displays images and text information. The display device 150 may be a display device separate from the other components of the control device 10.

[0021] The input device 160 is a device that accepts operations from a user of the control device 10. The user uses the input device 160 to input a first material image MI1 to be stored in the material image database MDB to the control device 10. The first material image MI1 input by the user is stored in the material image database MDB. For example, the input device 160 is configured to include a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 160 is configured to include a touch panel, it may also serve as the display device 150.

[0022] The communication device 170 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 170 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 170 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 170 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include those compliant with wired LAN (Local Area Network), IEEE 1394, and USB (Universal Serial Bus). Examples of the wireless communication interface include those compliant with wireless LAN, Bluetooth (registered trademark), etc.

[0023] The processing device 120 reads and executes the control program PR1 from the storage device 140, thereby functioning as a calibration unit 121, a material extraction unit 122, a projection target extraction unit 123, a matching processing unit 124, a geometric correction unit 125, a projection image generation unit 126, and a projection control unit 127. The control program PR1 may be transmitted from another device, such as a server that manages the control device 10, via a communication network.

[0024] The calibration unit 121 executes a calibration process between the imaging device 110 and the projector 20. When the calibration unit 121 executes the calibration process, for example, a pattern image is projected from the projector 20 onto a screen. For example, a gray code pattern image or a pattern image using a sine wave grayscale image is used as the pattern image. Thereafter, the imaging device 110 captures the pattern image projected onto the screen. The calibration unit 121 calculates pixel correspondence information indicating the correspondence between the pixels of the projector 20 and the pixels of the imaging device 110 based on the pattern image projected from the projector 20 and the pattern image captured by the imaging device 110. The calibration unit 121 outputs the pixel correspondence information to the projection image generation unit 126. Furthermore, based on the pixel correspondence information, the calibration unit 121 calculates depth information of the space that is the image capture target of the imaging device 110. The calibration unit 121 outputs the depth information to the projection target extraction unit 123.

[0025] The imaging device 110 captures an image of an object installed in the space, onto which the projector 20 projects a projection image PI corresponding to the material image MI, as will be described later. The calibration unit 121 outputs a first object image OI1, which is an image of the object captured by the imaging device 110, to the projection target extraction unit 123.

[0026] The material extraction unit 122 acquires a first material image MI1 from the material image database MDB. The material extraction unit 122 also removes an image of a background region from the acquired first material image MI1 and extracts an image of only the region to be mapped to the object. The material extraction unit 122 outputs a second material image MI2 from which only the region to be mapped has been extracted to the matching processing unit 124.

[0027] The projection target extraction unit 123 acquires a first object image OI1 from the calibration unit 121. The projection target extraction unit 123 also removes an image of the background region from the acquired first object image OI1 and extracts an image of only the region of the object to be projected. The projection target extraction unit 123 outputs a second object image OI2 from which only the region of the object to be projected has been extracted to the matching processing unit 124.

[0028] The matching processing unit 124 extracts a plurality of feature points CP required for mapping from each of the second material image MI2 acquired from the material extraction unit 122 and the second object image OI2 acquired from the projection target extraction unit 123. Each of the plurality of feature points CP extracted from the second material image MI2 is an example of a first feature point CP1. Each of the plurality of feature points CP extracted from the second object image OI2 is an example of a second feature point CP2. The matching processing unit 124 generates a first feature map CM1 using the plurality of first feature points CP1. Similarly, the matching processing unit 124 generates a second feature map CM2 using the plurality of second feature points CP2. Furthermore, the matching processing unit 124 obtains corresponding point information RI indicating the correspondence relationship between corresponding points RP among the plurality of first feature points CP1 included in the first feature map CM1 and the plurality of second feature points CP2 of the second object image OI2 included in the second feature map CM2. More specifically, the matching processing unit 124 derives corresponding point information RI indicating the correspondence relationship between a plurality of first corresponding points RP1 included in the plurality of first feature points CP1 and a plurality of second corresponding points RP2 included in the plurality of second feature points CP2. The matching processing unit 124 outputs the corresponding point information RI to the geometric correction unit 125. The matching processing unit 124 will be described in detail later.

[0029] The geometric correction unit 125 generates a third material image MI3 by performing geometric correction on the second material image MI2 using the corresponding point information RI acquired from the matching processing unit 124 so that the coordinates of the first corresponding point RP1 in the second material image MI2 and the coordinates of the second corresponding point RP2 in the second object image OI2 match. The geometric correction unit 125 outputs the generated third material image MI3 to the projection image generation unit 126 as a projection image PI.

[0030] The projection image generation unit 126 converts the coordinate system of the third material image MI3 acquired from the geometric correction unit 125 into a panel coordinate system used in the projector 20 based on the pixel correspondence information acquired from the calibration unit 121, thereby generating a projection image PI whose viewpoint is a projection device 210 (described later) provided in the projector 20. The projection image generation unit 126 outputs the generated projection image PI to the projection control unit 127.

[0031] The projection control unit 127 causes the projector 20 to project the projection image PI acquired from the projection image generation unit 126 onto the above-mentioned object.

[0032] 1-1-2-2: Features of the matching processing unit A: Input / output specifications FIG. 3 is a diagram showing an example of the specifications of information input and output to the matching processing unit 124. As described above, the matching processing unit 124 receives the second material image MI2 and the second object image OI2 as examples of two visually different images. As an example, the second material image MI2 and the second object image OI2 differ in at least one of local shape and color. As another example, the second material image MI2 and the second object image OI2 are each a color image, a gray image, or an edge image. As another example, the resolution of each of the second material image MI2 and the second object image OI2 is 3840 pixels × 2160 pixels. The matching processing unit 124 extracts feature points CP required for fitting between both images and obtains corresponding point information RI indicating corresponding points RP. As shown in FIG. 3, the corresponding point information RI is information that associates the coordinates of a first corresponding point RP1 included in a first feature point CP1 extracted from the second material image MI2 with the coordinates of a second corresponding point RP2 included in a second feature point CP2 extracted from the second object image OI2. In the example shown in FIG. 3, 13 pairs of coordinates of the first corresponding point RP1 included in the second material image MI2 and the coordinates of the second corresponding point RP2 included in the second object image OI2 are listed. However, the number of pairs of coordinates of the two corresponding points RP may be any number. Deep learning is used as a method for extracting the first feature point CP1 and the second feature point CP2 and deriving the corresponding point information RI. The details of this deep learning will be described later.

[0033] B: Outline of the configuration of the matching processing unit 4 to 10, the configuration of the matching processing unit 124 will be outlined below. In the following description, seven configurations of the matching processing unit 124 will be described: (b1) a configuration in which the feature point extraction function and the matching function are implemented by a single CNN (Convolutional Neural Network); (b2) a configuration in which a CNN is assigned to each of the feature point extraction function and the matching function; (b3) a configuration in which a mechanism for transmitting the features of each layer to a discrimination layer is added to a CNN that implements the feature point extraction function; (b4) a configuration in which a learning model for image classification is used to a CNN that implements the feature point extraction function; (b5) a configuration in which a feature pyramid mechanism (FPN: Feature Pyramid Network) is added to a CNN that implements the feature point extraction function; (b6) a configuration in which a matching mechanism for each resolution (multi-scale matching networks) is added to a CNN that implements the matching function; and (b7) a configuration in which an attention mechanism is added to a CNN that implements the feature point extraction function. However, these are merely examples of the configuration of the matching processing unit 124. The matching processing unit 124 may have a configuration other than the seven configurations described above.

[0034] b1: A configuration that realizes feature point extraction and matching functions with a single CNN 4 is a diagram showing an example of a configuration in which the matching processing unit 124 realizes the feature point extraction function and the matching function using a single CNN. In FIG. 4, the matching processing unit 124 includes a single CNN 124a. A combined image BI obtained by combining a second material image MI2 and a second object image OI2 is input to the single CNN 124a. The single CNN 124a performs a series of convolution operations to extract a plurality of first feature points CP1 from the second material image MI2 included in the combined image BI, extract a plurality of second feature points CP2 from the second object image OI2 included in the combined image BI, and derive correspondence point information RI indicating the correspondence between a plurality of first corresponding points RP1 included in the plurality of first feature points CP1 and a second corresponding point RP2 included in the plurality of second feature points CP2.

[0035] Details of the configuration example shown in FIG. 4 will be described later in the section "D: Detailed configuration of the matching processing unit."

[0036] b2: Configuration in which CNNs are assigned to both the feature point extraction function and the matching function FIG. 5 is a diagram illustrating an example of a configuration in which CNNs are assigned to each of the feature point extraction function and the matching function in the matching processing unit 124. In FIG. 5, the matching processing unit 124 includes a first CNN 124b[1], a second CNN 124b[2], and a third CNN 124b[3]. The first CNN 124b[1] extracts a plurality of first feature points CP1 by performing a first convolution operation on the second material image MI2. The second CNN 124b[2] extracts a plurality of second feature points CP2 by performing a second convolution operation on the second object image OI2. The third CNN 124b[3] derives corresponding point information RI indicating the correspondence between a plurality of first corresponding points RP1 included in the plurality of first feature points CP1 and a plurality of second corresponding points RP2 included in the plurality of second feature points CP2.

[0037] By separately providing a first CNN 124b[1] for performing a first convolution operation on the second material image MI2 and a second CNN 124b[2] for performing a second convolution operation on the second object image OI2, the first feature points CP1 and the second feature points CP2 are extracted by different CNNs. As a result, the accuracy of the correspondence indicated by the corresponding point information RI is improved.

[0038] Details of the configuration example shown in FIG. 5 will be described later in the section "D: Detailed configuration of matching processing unit."

[0039] b3: A structure in which a mechanism for transmitting the features of each layer to the classification layer is added to the CNN that realizes the feature point extraction function. FIG. 6 shows an example configuration of the matching processing unit 124, in which a CNN is assigned to each of the feature point extraction function and the matching function, and a mechanism is added to transmit the features of each layer of the CNN that realizes the feature point extraction function to the discrimination layer. Here, the "discrimination layer" refers to the CNN that realizes the matching function. In FIG. 6, the matching processing unit 124 includes a first CNN 124c[1], a second CNN 124c[2], and a third CNN 124c[3]. The first CNN 124c[1] extracts multiple first feature points CP1 by performing a first convolution operation on the second material image MI2. The second CNN 124c[2] extracts multiple second feature points CP2 by performing a second convolution operation on the second object image OI2. The third CNN 124c[3] derives corresponding point information RI indicating the correspondence relationship between a plurality of first corresponding points RP1 included in the plurality of first feature points CP1 and a plurality of second corresponding points RP2 included in the plurality of second feature points CP2.

[0040] The first CNN 124c[1] also includes a first sub-CNN 124c

[11] as a first encoder and a first sub-CNN 124c

[12] as a first decoder. The second material image MI2 is input to the first sub-CNN 124c

[11] and encoded, and then input to the first sub-CNN 124c

[12] and decoded. A skip connection is made to output encoded data from one of the multiple layers constituting the first sub-CNN 124c

[0111] to one of the multiple layers constituting the first sub-CNN 124c

[12] .

[0041] The first CNN 124c[1] includes the first sub-CNN 124c

[11] as a first encoder, so that the first feature map CM1 of the second material image MI2 is encoded and the dimension is reduced. Also, the first CNN 124c[1] includes the first sub-CNN 124c

[12] as a first decoder, so that the encoded first feature map CM1 is decoded.

[0042] Similarly, the second CNN 124c[2] includes a second sub-CNN 124c

[21] as a second encoder and a second sub-CNN 124c

[22] as a second decoder. The second object image OI2 is input to the second sub-CNN 124c

[21] and encoded, and then input to the second sub-CNN 124c

[22] and decoded. A skip connection is established between the second sub-CNN 124c

[21] and the second sub-CNN 124c

[22] , outputting encoded data from one of the layers constituting the second sub-CNN 124c

[21] to one of the layers constituting the second sub-CNN 124c

[22] .

[0043] The second CNN 124c[2] includes a second sub-CNN 124c

[21] as a second encoder, which encodes the second feature map CM2 of the second object image OI2 and reduces the dimension. The second CNN 124c[2] also includes a second sub-CNN 124c

[22] as a second decoder, which decodes the encoded second feature map CM2.

[0044] When simply classifying images, there is no particular problem if the feature maps CM in the deep layers are low resolution. However, when detecting images or matching feature points CP as in this embodiment, low resolution feature maps CM in the deep layers becomes a problem. Therefore, when matching feature points CP, it is necessary to transmit high-resolution feature maps CM in the shallow and intermediate layers up to the classification layer.

[0045] In the above configuration example, the first CNN 124c[1] and the second CNN 124c[2] each include an encoder and a decoder. The encoder performs downsampling, which involves convolution and pooling, on the input image multiple times to extract a low-resolution feature map CM specific to the input image. The decoder performs upsampling, which involves deconvolution, multiple times to increase the resolution of the feature map CM specific to the input image. However, simply performing upsampling reduces the accuracy of the position information of objects contained in the input image. Therefore, skip connections are used to connect the same-scale layers of the encoder and decoder. As a result, information from the high-resolution feature map CM in the middle layer is transmitted from the shallow layer of the encoder to the decoder, enabling highly accurate upsampling of object position information. This structure allows low-level features in the shallow layer (e.g., features such as object edges) and mid-level features in the middle layer (e.g., features such as object parts) to be transmitted to the classification layer without losing the information on the features themselves or their position information. As a result, highly accurate matching of feature points CP is possible.

[0046] Details of the configuration example shown in FIG. 6 will be described later in the section "D: Detailed configuration of the matching processing unit."

[0047] b4: A configuration that uses a learning model for image classification in a CNN that realizes feature point extraction function FIG. 7 is a diagram illustrating an example of a configuration in which a learning model for image classification is used in a CNN that realizes a feature point extraction function in the matching processing unit 124. In FIG. 7, the matching processing unit 124 includes a first CNN 124d[1], a second CNN 124d[2], and a third CNN 124d[3]. The first CNN 124d[1] extracts multiple first feature points CP1 by performing a first convolution operation on the second material image MI2. The second CNN 124d[2] extracts multiple second feature points CP2 by performing a second convolution operation on the second object image OI2. The third CNN 124d[3] derives corresponding point information RI indicating the correspondence between multiple first corresponding points RP1 included in the multiple first feature points CP1 and multiple second corresponding points RP2 included in the multiple second feature points CP2.

[0048] Furthermore, each of the first CNN 124d[1] and the second CNN 124d[2] is a learning model for image classification. As an example, each of the first CNN 124d[1] and the second CNN 124d[2] may be a VGG (Visual Geometry Group) network. More specifically, each of the first CNN 124d[1] and the second CNN 124d[2] may be a 16-layer VGG network or a 19-layer VGG network.

[0049] The learning model for image classification has already learned features similar to those used to extract feature points CP. By performing additional learning using training data while the learning model for image classification is used as the first CNN 124d[1] and the second CNN 124d[2], it becomes possible to learn with training data that has a smaller number of data sets than would have been required.

[0050] b5: A configuration in which a feature pyramid mechanism (FPN: Feature Pyramid Network) is added to a CNN that realizes feature point extraction function. FIG. 8 illustrates an example configuration of the matching processor 124, in which a feature pyramid mechanism (FPN) is added to the CNN that realizes the feature point extraction function. The feature pyramid mechanism uses a multi-scale CNN as the base structure as the encoder, and then combines a multi-scale CNN as the decoder in the latter half of the base structure using skip connections for each scale. As a result, the feature pyramid mechanism can learn features at various scales as a result of passing through various scale paths. In this embodiment, the feature pyramid mechanism executes a process to extract feature points CP for each resolution. The addition of the feature pyramid mechanism enables the CNN in this embodiment to calculate a feature map that includes all features at multiple image scales, thereby increasing robustness in image recognition tasks that must adapt to the magnitude of relative scale changes within an image. As a result, for example, the accuracy of identifying an object of a specific size from a photograph containing objects of various sizes is improved. In FIG. 8, the matching processor 124 includes a first CNN 124e[1], a second CNN 124e[2], and a third CNN 124e[3]. The first CNN 124e[1] extracts a plurality of first feature points CP1 by performing a first convolution operation on the second material image MI2. The second CNN 124e[2] extracts a plurality of second feature points CP2 by performing a second convolution operation on the second object image OI2. The third CNN 124e[3] derives corresponding point information RI indicating the correspondence between a plurality of first corresponding points RP1 included in the plurality of first feature points CP1 and a plurality of second corresponding points RP2 included in the plurality of second feature points CP2.

[0051] The first CNN124e[1] also includes a first sub-CNN124e

[11] as a learning model for image classification and a first sub-CNN124e

[12] as a feature pyramid mechanism. After the second material image MI2 is input to the first sub-CNN124e

[12] , the first feature point CP1 is extracted by inputting the second material image MI2 to the first sub-CNN124e

[12] .

[0052] The second CNN 124e[2] includes a second sub-CNN 124e

[21] as a learning model for image classification and a second sub-CNN 124e

[22] as a feature pyramid mechanism. The second object image OI2 is input to the second sub-CNN 124e

[21] and then input to the second sub-CNN 124e

[22] to extract second feature points CP2.

[0053] In the first CNN 124d[1], a feature pyramid structure is added after the learning model for image classification, so that the first CNN 124d[1] can extract feature points CP for each resolution of the second material image MI2. Also, in the second CNN 124d[2], a feature pyramid structure is added after the learning model for image classification, so that the second CNN 124d[2] can extract feature points CP for each resolution of the second object image OI2.

[0054] b6: A configuration in which a matching mechanism for each resolution (Multi-scale Matching Networks) is added to the CNN that realizes the matching function. FIG. 9 is a diagram illustrating an example configuration of the matching processing unit 124, in which a matching mechanism (MMNet) that performs matching processing for each resolution is added to the CNN that realizes the matching function. In FIG. 9, the matching processing unit 124 includes a first CNN 124f[1], a second CNN 124f[2], and a third CNN 124f[3]. The first CNN 124f[1] extracts multiple first feature points CP1 by performing a first convolution operation on the second material image MI2. The second CNN 124f[2] extracts multiple second feature points CP2 by performing a second convolution operation on the second object image OI2. The third CNN 124f[3] derives corresponding point information RI indicating the correspondence between multiple first corresponding points RP1 included in the multiple first feature points CP1 and multiple second corresponding points RP2 included in the multiple second feature points CP2.

[0055] The first CNN124f[1] also includes a first sub-CNN124f

[11] as a learning model for image classification and a first sub-CNN124f

[12] as a feature pyramid mechanism. After the second material image MI2 is input to the first sub-CNN124f

[12] , the first feature point CP1 is extracted by inputting it to the first sub-CNN124f

[12] .

[0056] The second CNN 124f[2] includes a second sub-CNN 124f

[21] as a learning model for image classification and a second sub-CNN 124f

[22] as a feature pyramid mechanism. The second object image OI2 is input to the second sub-CNN 124f

[21] and then input to the second sub-CNN 124f

[22] to extract second feature points CP2.

[0057] The third CNN 124f[3] also includes a per-resolution matching mechanism 124f

[31] . By including the matching mechanism 124f

[31] in the third CNN 124f[3], matching processing is performed for each resolution of the second material image MI2 input to the first CNN 124f[1] and for each resolution of the second object image OI2 input to the second CNN 124f[2]. By including the per-resolution matching mechanism 124f

[31] in the third CNN 124f[3], for example, the matching result of the previous scale is directly upsampled and combined with the details of the current matching to learn complementary correspondences between scales.

[0058] b7: A CNN that realizes feature point extraction function is equipped with an attention mechanism FIG. 10 is a diagram illustrating an example of a configuration in which a CNN that realizes a feature point extraction function in the matching processing unit 124 includes an attention mechanism. The attention mechanism is a mechanism that dynamically identifies which part of input data should be focused on. A CNN with an attention mechanism can, for example, remove background that is irrelevant to image recognition and then focus on the foreground for learning during image recognition. In FIG. 10, the matching processing unit 124 includes a first CNN 124g[1], a second CNN 124g[2], and a third CNN 124g[3]. The first CNN 124g[1] extracts multiple first feature points CP1 by performing a first convolution operation on the second material image MI2. The second CNN 124g[2] extracts multiple second feature points CP2 by performing a second convolution operation on the second object image OI2. The third CNN 124g[3] derives corresponding point information RI indicating the correspondence between a plurality of first corresponding points RP1 included in the plurality of first feature points CP1 and a plurality of second corresponding points RP2 included in the plurality of second feature points CP2.

[0059] In addition, each of the first CNN 124g[1] and the second CNN 124g[2] is equipped with an attention mechanism. By having the attention mechanism in the first CNN 124g[1], it becomes possible to learn in a state where the background region is removed from the material image MI as training data. In addition, by having the attention mechanism in the second CNN 124g[2], it becomes possible to learn in a state where the background region is removed from the object image OI as training data.

[0060] The third CNN 124g[3] also includes a matching mechanism 124g

[31] for each resolution. By including the matching mechanism 124g

[31] in the third CNN 124g[3], matching processing is performed for each resolution of the second material image MI2 input to the first CNN 124g[1] and for each resolution of the second object image OI2 input to the second CNN 124g[2].

[0061] C: Training dataset creation method and model resolution For the sake of simplicity, the following will explain the training data for the CNN that realizes the above matching function, taking the above-mentioned "(b2) Configuration in which a CNN is assigned to each of the feature point extraction function and the matching function" as an example.

[0062] In this embodiment, the third CNN 124b[3] learns, in the learning stage, correspondences between a plurality of first learning corresponding points RP1 and a plurality of second learning corresponding points RP2 as training data. The plurality of first learning corresponding points RP1 are included in a plurality of first learning feature points CP1. The plurality of first learning feature points CP1 are extracted by inputting, to the first CNN 124b[1], a second-resolution material image MI obtained by reducing a first-resolution material image MI at a predetermined reduction ratio. The plurality of second learning corresponding points RP2 are included in a plurality of second learning feature points CP2. The plurality of second learning feature points CP2 are extracted by inputting, to the second CNN 124b[2], a second-resolution object image OI obtained by reducing a first-resolution object image OI at a predetermined reduction ratio. Here, the first resolution is greater than the second resolution. In addition, in the operation stage, a second material image MI2 with a second resolution is input to the first CNN 124b[1], and a second object image OI2 with a second resolution is input to the second CNN 124b[2].

[0063] When creating the dataset, as an example, the worker first prepares original images of 2160 pixels x 3840 pixels. For example, the worker prepares 74 original images of 2160 pixels x 3840 pixels as material images MI and 24 original images of 2160 pixels x 3840 pixels as object images OI. Next, the worker visually sets the coordinate positions of feature points CP for these images. At this stage, the coordinate positions of feature points CP are expressed as pixel numerical values. These numerical values ​​are integers.

[0064] Next, the worker reduces the original image to various magnifications. For example, the worker creates an image in which the original image is reduced to half horizontally and half vertically, an image in which the original image is reduced to half horizontally and a quarter vertically, an image in which the original image is reduced to a quarter horizontally and an eighth vertically, and an image in which the original image is reduced to an eighth horizontally and a sixteenth vertically. In these images, the coordinate positions of the feature points CP are expressed as decimals according to the reduction magnification.

[0065] Finally, the worker creates a total of 1008 sets of data sets as training data by combining 56 material images MI and 18 object images OI for each resolution. More specifically, as an example, the worker selects 56 of the 74 material images MI of the original images and 18 of the 24 object images OI of the original images, and combines these to create a total of 1008 sets of data sets as training data. In these data sets, the coordinate positions of the feature points CP are expressed as decimals according to the reduction ratio, as described above.

[0066] The third CNN 124b[3] has a learning model that uses a dataset of training data consisting of a combination of a material image MI, in which the coordinate position of the first feature point CP1 is expressed as a decimal, and an object image OI, in which the coordinate position of the second feature point CP2 is expressed as a decimal. Meanwhile, in the operation phase, the coordinate position of the first feature point CP1 in the second material image MI2 input to the third CNN 124b[3] and the coordinate position of the second feature point CP2 in the second object image OI2 are expressed as integers. Furthermore, the resolution of the original images of the material image MI and object image OI used in the dataset as training data is greater than the resolution of the second material image MI2 input to the first CNN 124b[1] and the resolution of the second object image OI2 input to the second CNN 124b[2] in the operation phase. Therefore, the third CNN 124b[3] can generate highly accurate correspondences. That is, instead of using the high-resolution second material image MI2 and object image OI as training data as they are, the resolution of the second material image MI2 and object image OI is converted into a pseudo-low resolution at a predetermined reduction ratio, and a learning model is constructed using the pseudo-low-resolution second material image MI2 and object image OI. In this way, the learning model can output the coordinates of the first feature point CP1 and the coordinates of the second feature point CP2 as decimals, which could previously only be expressed as integers, thereby achieving high accuracy in the correspondence relationship. Note that the lower the reduction ratio of the material images MI and object images OI used in the training data set, the greater the number of data sets required for training and the longer the training time. Therefore, in practice, the third CNN 124b[3] that uses as training data a data set generated using a combination of material images MI and object images OI that have been reduced to a smaller value at a reduction ratio equal to or greater than a predetermined value can perform matching with higher accuracy.

[0067] In the above explanation, "(b2) a configuration in which a CNN is assigned to each of the feature point extraction function and the matching function" has been used as an example. However, the same applies to other cases, such as "(b3) a configuration in which a mechanism for transmitting the features of each layer to the identification layer is added to a CNN that realizes the feature point extraction function," "(b4) a configuration in which a learning model for image classification is used to a CNN that realizes the feature point extraction function," "(b5) a configuration in which a feature pyramid mechanism (FPN) is added to a CNN that realizes the feature point extraction function," "(b6) a configuration in which a matching mechanism for each resolution (MMNet) is added to a CNN that realizes the matching function," and "(b7) a configuration in which an attention mechanism is added to a CNN that realizes the feature point extraction function."

[0068] D: Details of the matching processing unit configuration 11 to 17, the configuration of the matching processing unit 124 will be described in detail below. Note that the following describes the details of the above "(b1) configuration in which the feature point extraction function and the matching function are realized by a single CNN (Convolutional Neural Network)", "(b2) configuration in which a CNN is assigned to each of the feature point extraction function and the matching function", and "(b3) configuration in which a mechanism for transmitting the features of each layer to the identification layer is added to the CNN that realizes the feature point extraction function". In the following, we will mainly examine the details of the components and hyperparameters of "(b2) Configuration in which CNNs are assigned to the feature point extraction function and the matching function respectively" together with the verification results.

[0069] d1: A configuration that realizes feature point extraction and matching functions with a single CNN FIG. 11 is a diagram showing an example of the configuration of the matching processing unit 124 in a configuration in which the feature point extraction function and the matching function are realized by a single CNN.

[0070] The matching processing unit 124 in this configuration example includes the CNN 124a as described above. A combined image BI obtained by combining the second material image MI2 and the second object image OI2 is input to the CNN 124a. Furthermore, when each block constituting the CNN 124a is counted as one layer, the CNN 124a has a five-layer structure, including a first layer La1 to a fifth layer La5, as shown in FIG. 11 . The number of channels in the convolutional layer included in the first layer La1 is 128. The number of channels in the convolutional layer included in the second layer La2 is 64. The number of channels in the convolutional layer included in the third layer La3 is 32. The number of channels in the convolutional layer included in the fourth layer La4 is 16. As such, in the CNN 124a, the number of channels in each convolutional layer decreases continuously or stepwise from the input-side convolutional layer to the output-side convolutional layer. That is, in the CNN 124a, the number of channels in the output-side convolutional layer is smaller than the number of channels in the input-side convolutional layer. Note that these numbers of channels are merely examples. Furthermore, these numbers of channels are multiples of the number of first corresponding points RP1 and the number of second corresponding points RP2 included in the corresponding point information RI output from the CNN 124a.

[0071] As described above, the CNN 124a has a five-layer structure, and the number of channels in each convolutional layer decreases continuously or stepwise from the input-side convolutional layer to the output-side convolutional layer.

[0072] d2: Configuration in which CNNs are assigned to both the feature point extraction function and the matching function d2-1: Configuration details FIG. 12 is a diagram showing an example of the configuration of the matching processing unit 124 in a configuration in which CNNs are assigned to both the feature point extraction function and the matching function.

[0073] As described above, the matching processing unit 124 in this configuration example includes a first CNN 124b[1], a second CNN 124b[2], and a third CNN 124b[3]. A second material image MI2 is input to the first CNN 124b[1]. A second object image OI2 is input to the second CNN 124b[1].

[0074] Furthermore, when each block constituting the first CNN 124b[1] is counted as one layer, the first CNN 124b[1] has a four-layer structure consisting of the first layer Lb1[1] to the fourth layer Lb4[1], as shown in FIG. 12. The number of channels in the convolutional layer included in the first layer Lb1[1] is 138. The number of channels in the convolutional layer included in the second layer Lb2[1] is 92. The number of channels in the convolutional layer included in the third layer Lb3[1] is 69. The number of channels in the convolutional layer included in the fourth layer Lb4[1] is 46. Thus, in the first CNN 124b[1], the number of channels in each convolutional layer decreases continuously or stepwise from the input-side convolutional layer to the output-side convolutional layer. That is, in the first CNN 124b[1], the number of channels in the output-side convolutional layer is smaller than the number of channels in the input-side convolutional layer. Note that these numbers of channels are merely examples. Furthermore, these numbers of channels are multiples of the number of first corresponding points RP1 included in the corresponding point information RI output from the third CNN 124c.

[0075] Furthermore, when each block constituting the second CNN 124b[2] is counted as one layer, the second CNN 124b[2] has a four-layer structure consisting of the first layer Lb1[2] to the fourth layer Lb4[2], as shown in FIG. 12. The number of channels in the convolutional layer included in the first layer Lb1[2] is 138. The number of channels in the convolutional layer included in the second layer Lb2[2] is 92. The number of channels in the convolutional layer included in the third layer Lb3[2] is 69. The number of channels in the convolutional layer included in the fourth layer Lb4[2] is 46. Thus, in the second CNN 124b[2], the number of channels in each convolutional layer decreases continuously or stepwise from the input-side convolutional layer to the output-side convolutional layer. That is, in the second CNN 124b[2], the number of channels in the output-side convolutional layer is smaller than the number of channels in the input-side convolutional layer. Note that these numbers of channels are merely examples. Furthermore, these numbers of channels are multiples of the number of second corresponding points RP2 included in the corresponding point information RI output from the third CNN 124c.

[0076] Furthermore, when each block constituting the third CNN 124b is counted as one layer, the third CNN 124b has a single-layer structure of a fifth layer Lb5 as shown in FIG. That is, the matching processing unit 124 as a whole has a five-layer structure as a CNN.

[0077] Furthermore, a normalization layer is applied only immediately after the input layer (not shown in FIG. 12) provided in each of the first CNN 124b[1] and the second CNN 124b[2].

[0078] It is preferable to reduce the kernel size, which is one of the hyperparameters used in the matching processing unit 124, and instead increase the number of layers in the entire CNN in the entire matching processing unit 124.

[0079] Furthermore, when optimizing the details of the configuration of the matching unit 124, it is preferable to perform dropout last. Furthermore, when optimizing the hyperparameters used in the matching unit 124, it is preferable to perform the regularization process last.

[0080] In addition, it is preferable to use ReLu(max=n) as the activation function of the output layer included in the third CNN 124b[3] instead of the common Linear. Note that "n" here is the maximum resolution of the second material image MI2 input to the first CNN 124b[1] and the maximum resolution of the second object image OI2 input to the second CNN 124b[2] during operation.

[0081] d2-2: Contents of the study Below, we will explain some of the details of the configuration in which CNNs are assigned to each of the feature point extraction function and the matching function, and the reasons for the above settings for some of the hyperparameters, with reference to Figures 13 to 16. During the study, the worker input a second material image MI2 of 256 pixels x 256 pixels to the first CNN 124b[1], and input a second object image OI2 of 256 pixels x 256 pixels to the second CNN 124b[2].

[0082] Then, the average positional deviation (hereinafter referred to as the "average positional deviation") between the correct values ​​determined by visual inspection of the feature point positions of the second material image MI2 and the second object image OI2 and the estimated values ​​of the feature point positions of each when the control device 10 superimposes the second material image MI2 on the second object image OI2 was used as a measure of the accuracy of the matching processing unit 124.

[0083] More specifically, for the control device 10, the following items were examined by using 7104 sets of data, or 20% of the total 35,520 sets of data, which were a combination of 74 raw images MI of the original image and 480 images in which 24 object images OI of the original image were data-expanded 20 times.

[0084] d2-2-1: Number of layers FIG. 13 is a graph showing the relationship between the number of CNN layers in the entire matching processing unit 124 and the average positional deviation. The horizontal axis represents the number of CNN layers, and the vertical axis represents the average positional deviation. As shown in FIG. 13, when the number of CNN layers was five, the average positional deviation was minimized. In determining the number of CNN layers, a policy was adopted in which the number of layers was increased from two to the maximum number possible while still improving performance, and therefore, as described above, the number of CNN layers in the entire matching processing unit 124 was set to five.

[0085] d2-2―2: Number of channels FIG. 14 is a graph showing the relationship between the number of channels in each hidden layer in the matching processing unit 124 and the average positional deviation. The horizontal axis represents the number of channels in each hidden layer. Each item on the horizontal axis is a combination of four numbers. From left to right, the four numbers represent the number of channels in the first layer, the number of channels in the second layer, the number of channels in the third layer, and the number of channels in the fourth layer. The vertical axis represents the average positional deviation. Among the combinations of the number of channels shown in FIG. 14, the average positional deviation was smallest when the combinations of the number of channels in the first layer, the number of channels in the second layer, the number of channels in the third layer, and the number of channels in the fourth layer were 138 channels, 92 channels, 69 channels, and 46 channels, respectively. As shown in FIG. 14, it was shown that a configuration in which the number of channels is large in the first hidden layer and decreases toward the last layer is preferable.

[0086] d2-2―3: Normalization layer Figure 15 is a graph showing the relationship between the presence and position of a normalization layer and the average positional deviation. The horizontal axis indicates the presence and position of a normalization layer, and the vertical axis indicates the average positional deviation. As shown in Figure 15, when a normalization layer was provided only immediately after the input layer, the average positional deviation was minimized.

[0087] d2-2-4: Kernel size FIG. 16 is a graph showing the relationship between kernel size and average positional deviation. The horizontal axis shows the length of each vertical and horizontal side of the square kernel. The vertical axis shows the average positional deviation. As shown in FIG. 15, when the kernel size was 3 pixels x 3 pixels, the average positional deviation was minimized.

[0088] d3: A structure in which a mechanism for transmitting the features of each layer to the classification layer is added to the CNN that realizes the feature point extraction function. FIG. 17 is a diagram showing an example of the configuration of the matching processing unit 124 in a configuration in which a mechanism for transmitting the features of each layer to the discrimination layer is added to the CNN that realizes the feature point extraction function.

[0089] As described above, the matching processing unit 124 in this configuration example includes a first CNN 124c[1], a second CNN 124c[2], and a third CNN 124c[3]. A second material image MI2 is input to the first CNN 124c[1]. A second object image OI2 is input to the second CNN 124c[1].

[0090] As described above, the first CNN 124c[1] includes the first sub-CNN 124c

[11] as the first encoder and the first sub-CNN 124c

[12] as the first decoder. The second material image MI2 is input to the first sub-CNN 124c

[11] and encoded, and then input to the first sub-CNN 124c

[12] and decoded.

[0091] Furthermore, when each block constituting the first sub-CNN 124c

[11] is counted as one layer, the first sub-CNN 124c

[11] has a four-layer structure consisting of the first layer Lc1

[11] to the fourth layer Lc4

[11] , as shown in FIG. 17. The number of channels in the convolutional layer included in the first layer Lc1

[11] is 16. The number of channels in the convolutional layer included in the second layer Lc2

[11] is 32. The number of channels in the convolutional layer included in the third layer Lc3

[11] is 64. The number of channels in the convolutional layer included in the fourth layer Lc4

[11] is 128. In this way, in the first sub-CNN 124c

[11] , the number of channels in each convolutional layer increases continuously or stepwise from the input convolutional layer to the output convolutional layer.

[0092] Furthermore, when each block constituting the first sub-CNN 124c

[12] is counted as one layer, the first sub-CNN 124c

[12] has a four-layer structure consisting of the first layer Lc1

[12] to the fourth layer Lc4

[12] as shown in FIG. 17. The number of channels in the convolutional layer included in the second layer Lc2

[12] is 64. The number of channels in the convolutional layer included in the third layer Lc3

[12] is 32. The number of channels in the convolutional layer included in the fourth layer Lc4

[12] is 16. Thus, in the first sub-CNN 124c

[12] , the number of channels in each convolutional layer decreases continuously or stepwise from the input-side convolutional layer to the output-side convolutional layer. That is, in the first sub-CNN 124c

[12] , the number of channels in the output-side convolutional layer is smaller than the number of channels in the input-side convolutional layer.

[0093] The first CNN 124c[1] includes a fifth layer Lc5[1] as a further block. The second material image MI2 is input to the first sub-CNN 124c

[11] , which functions as a first encoder. Information output from the first sub-CNN 124c

[11] is input to the first sub-CNN 124c

[12] , which functions as a first decoder. Information output from the first sub-CNN 124c

[12] is input to the fifth layer Lc5[1]. A first feature map CM1 consisting of first feature points CP1 is output from the fifth layer Lc5[1]. The fifth layer Lc5[1] includes one convolutional layer channel.

[0094] In the first CNN 124c[1], a skip connection is made from the first layer Lc1

[11] of the first sub-CNN 124c

[11] to the fourth layer Lc4

[12] of the first sub-CNN 124c

[12] . Information encoded by the first layer Lc1

[11] is output to the fourth layer Lc4

[12] and decoded in the fourth layer Lc4

[12] . In addition, a skip connection is made from the second layer Lc2

[11] of the first sub-CNN 124c

[11] to the third layer Lc3

[12] of the first sub-CNN 124c

[12] . The information encoded by the second layer Lc2

[11] is output to the third layer Lc3

[12] and decoded in the third layer Lc3

[12] . In addition, a skip connection is made from the third layer Lc3

[11] of the first sub-CNN 124c

[11] to the second layer Lc2

[12] of the first sub-CNN 124c

[12] . The information encoded by the third layer Lc3

[11] is output to the second layer Lc2

[12] and decoded by the second layer Lc2

[12] .

[0095] The second CNN 124c[2] includes a second sub-CNN 124c

[21] as a second encoder and a second sub-CNN 124c

[22] as a second decoder. The second object image OI2 is input to the second sub-CNN 124c

[21] and encoded, and then input to the second sub-CNN 124c

[22] and decoded.

[0096] Furthermore, when each block constituting the second sub-CNN 124c

[21] is counted as one layer, the second sub-CNN 124c

[21] has a four-layer structure consisting of the first layer Lc1

[21] to the fourth layer Lc4

[21] , as shown in FIG. 17. The number of channels in the convolutional layer included in the first layer Lc1

[21] is 16. The number of channels in the convolutional layer included in the second layer Lc2

[21] is 32. The number of channels in the convolutional layer included in the third layer Lc3

[21] is 64. The number of channels in the convolutional layer included in the fourth layer Lc4

[21] is 128. In this way, in the second sub-CNN 124c

[21] , the number of channels in each convolutional layer increases continuously or stepwise from the input convolutional layer to the output convolutional layer.

[0097] Furthermore, as shown in FIG. 12, when each block constituting the second sub-CNN 124c

[22] is counted as one layer, the second sub-CNN 124c

[22] has a four-layer structure consisting of the first layer Lc1

[22] to the fourth layer Lc4

[22] . The number of channels in the convolutional layer included in the second layer Lc2

[22] is 64. The number of channels in the convolutional layer included in the third layer Lc3

[22] is 32. The number of channels in the convolutional layer included in the fourth layer Lc4

[22] is 16. Thus, in the second sub-CNN 124c

[22] , the number of channels in each convolutional layer decreases continuously or stepwise from the input-side convolutional layer to the output-side convolutional layer. That is, in the second sub-CNN 124c

[22] , the number of channels in the output-side convolutional layer is smaller than the number of channels in the input-side convolutional layer.

[0098] The second CNN 124c[2] includes a fifth layer Lc5[2] as a further block. The second object image OI2 is input to the second sub-CNN 124c

[21] , which functions as a second encoder. Information output from the second sub-CNN 124c

[21] is input to the second sub-CNN 124c

[22] , which functions as a second decoder. Information output from the second sub-CNN 124c

[22] is input to the fifth layer Lc5[2]. A second feature point CP2 is output from the fifth layer Lc5[2]. The fifth layer Lc5[2] includes one convolutional layer channel.

[0099] In the second CNN 124c[2], a skip connection is made from the first layer Lc1

[21] of the second sub-CNN 124c

[21] to the fourth layer Lc4

[22] of the second sub-CNN 124c

[22] . Information encoded by the first layer Lc1

[21] is output to the fourth layer Lc4

[22] and decoded in the fourth layer Lc4

[22] . In addition, a skip connection is made from the second layer Lc2

[21] of the second sub-CNN 124c

[21] to the third layer Lc3

[22] of the second sub-CNN 124c

[22] . The information encoded by the second layer Lc2

[21] is output to the third layer Lc3

[22] and decoded by the third layer Lc3

[22] . Additionally, a skip connection is made from the third layer Lc3

[21] of the first sub-CNN 124c

[21] to the second layer Lc2

[22] of the second sub-CNN 124c

[22] . The information encoded by the third layer Lc3

[21] is output to the second layer Lc2

[22] and decoded by the second layer Lc2

[22] .

[0100] Furthermore, when each block constituting the third CNN 124c is counted as one layer, the third CNN 124c has a single-layer structure of a sixth layer Lc6 as shown in FIG. That is, the matching processing unit 124 as a whole has a six-layer structure as a CNN.

[0101] E: Data extension In order to improve the accuracy of matching by the matching processing unit 124 and to enable high-speed mapping for a variety of mapping targets in a variety of environments as a measure for environmental robustness, it is preferable to expand the number of data sets of training data used when training the CNN provided in the matching processing unit 124. As an example, it is possible to generate 10,000 first object images OI1 used as training data from one first object image OI1 and multiple mask images that mask the first object image OI1.

[0102] Specifically, data expansion is preferably achieved by generating images while varying environmental variables, including the relative positional relationship between the object and the camera included in the first object image OI1 to be mapped, the local shape of the object, the camera exposure, the camera contrast, the camera white balance, the camera's MTF (Modulation Transfer Function) characteristics, the brightness of the lighting, and the background of the object in the space in which the object is installed. As an example, data expansion can be achieved by generating various images while replacing the background of the object in the first object image OI1 with a background image of a plain, a group of buildings, mountains, etc. The "relative positional relationship" mentioned above includes the relative relationship regarding at least one of the relative horizontal position between the object and the camera, the relative vertical position between the object and the camera, the relative depth position between the object and the camera (the degree of magnification or reduction of the image captured by the camera), and the rotation angle of the camera around the roll axis relative to the object.

[0103] Furthermore, by expanding the number of data sets of training data while changing the local shape of the object, highly accurate matching becomes possible even when the object is an elastic body.

[0104] Furthermore, when expanding the dataset of training data while changing the above environmental variation factors, data expansion can be performed by performing an affine transformation that changes the relative positional relationship between the object and the camera, then synthesizing various backgrounds, and finally changing the brightness of the image. This increases the variety of combinations of geometric transformations between the background and the object compared to when the brightness of the image is changed after synthesizing the background and then finally performing an affine transformation, thereby enabling highly accurate matching. Alternatively, an affine transformation can be performed on an image of an object to change the shape of the object, and an affine transformation can be performed on a background image to change the shape of the background image, and the image of the object after the shape change can be synthesized with the background image after the shape change.

[0105] Furthermore, when synthesizing a background image, if the area near the boundary between the object and the background is set to a halftone depending on the area of ​​the background, the boundary position can be reproduced in a pseudo manner with sub-pixel accuracy, enabling highly accurate matching.

[0106] Furthermore, images captured by commercially available cameras may have a variety of aspect ratios. To achieve robust estimation for images with any aspect ratio, highly accurate matching is possible by inserting white space into the first object image OI1 and then performing data expansion. As an example, if the first object image OI1, which is 2656 pixels x 3984 pixels, is converted into a square image of 3984 pixels x 3984 pixels, then reduced to an image of 1288 pixels x 1228 pixels, and then reduced to an image of 2048 pixels x 2048 pixels by inserting white space, this image is then reduced to an image of 256 pixels x 256 pixels, enabling highly accurate matching regardless of the image aspect ratio.

[0107] The data extension for the first material image MI1 is similar to the above.

[0108] In addition, it is preferable to increase the number of data sets by changing the combination of the first material image MI1 and the first object image OI1 in the data sets as training data.

[0109] 1-1-3: Comparative example 1-1-3-1: First Comparative Example 18 is a diagram showing an algorithm executed by the matching processing unit 124A according to the first comparative example. The matching processing unit 124A executes the matching process by a single GMC (Geometric-matching CNN).

[0110] Specifically, as shown in Fig. 18, in step S1[1], the matching processing unit 124A extracts a first feature point CP1, which is a feature point CP of the material image MI. In step S1[2], which is parallel to step S1[1], the matching processing unit 124A extracts a second feature point CP2, which is a feature point CP of the object image OI. In step S2, the matching processing unit 124A matches the first feature point CP1 with the second feature point CP2. In step S3, the matching processing unit 124A calculates parameters for geometric transformation.

[0111] However, the accuracy of the alignment performed by the matching processing unit 124A is lower than the accuracy of the alignment performed by the matching processing unit 124 according to the present disclosure. The reason for this is that the data used for the matching process is the deepest layer data that has been subjected to Max Pooling processing four times, and the spatial resolution is 1 / 16, so the accuracy of estimating the positions of the corresponding points RP is low.

[0112] 1-1-3-2: Second Comparative Example 19 is a diagram showing an algorithm executed by the matching processing unit 124B according to the second comparative example. The matching processing unit 124B executes the matching process using a Parse Free Appearance Flow Network (PF-AFN).

[0113] Specifically, as shown in FIG. 19 , in step S11, the matching processing unit 124B performs preprocessing. Specifically, from a person image TI, the matching processing unit 124B estimates the pose of the person represented by the person image TI, which parts of the person image TI correspond to each feature of the person, and the processing target area in the person image TI. In step S12, the matching processing unit 124B performs layout estimation. Specifically, the matching processing unit 124B estimates a layout when the person represented by the person image TI is dressed in the clothes represented by the clothing image WI. In step S13, the matching processing unit 124B performs warping. Specifically, the matching processing unit 124B transforms the clothing image WI to achieve a target layout. In step S14, the matching processing unit 124B performs image generation. Specifically, the matching processing unit 124B generates an image in which the person represented by the person image TI is wearing the clothes represented by the clothing image WI.

[0114] However, the matching processing unit 124B can only generate an image in which the person represented by the person image TI is wearing the jacket represented by the clothing image WI. In other words, the matching processing unit 124B can only use an image of the jacket as the clothing image WI. This is because the segmentation processing performed by the matching processing unit 124B is specialized for jackets. Therefore, unlike the matching processing unit 124 according to the present disclosure, the matching processing unit 124B cannot be applied to objects other than jackets.

[0115] 1-2: Projector configuration 20 is a block diagram showing an example of the configuration of the projector 20. The projector 20 includes a projection device 210, a processing device 220, a storage device 230, and a communication device 240. The elements of the projector 20 are connected to one another by one or more buses for communicating information. Furthermore, the elements of the projector 20 are configured by one or more devices, and some elements of the projector 20 may be omitted.

[0116] The projection device 210 is a device that projects an image represented by an image signal acquired by an acquisition unit 221 (described later) onto a screen, a wall, or the like. The projection device 210 projects various images under the control of the processing device 220. The projection device 210 includes, for example, a light source, a liquid crystal panel, and a projection lens, and modulates light from the light source using the liquid crystal panel and projects the modulated light onto a screen, a wall, or the like via the projection lens.

[0117] The processing device 220 is a processor that controls the entire projector 20, and is configured, for example, by one or more chips. The processing device 220 is configured, for example, by a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 220 may be realized by hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 220 executes various processes in parallel or sequentially.

[0118] The storage device 230 is a recording medium readable by the processing device 220, and stores a plurality of programs including the control program PR2 executed by the processing device 220. The storage device 230 may be configured with at least one of, for example, a ROM, an EPROM, an EEPROM, and a RAM. The storage device 230 may also be called a register, a cache, a main memory, a primary storage device, or the like.

[0119] The communication device 240 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 240 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 240 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 240 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include those compliant with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include those compliant with wireless LAN, Bluetooth (registered trademark), etc.

[0120] The processing device 220 reads out and executes the control program PR2 from the storage device 230, thereby functioning as an acquisition unit 221 and a projection control unit 222. The control program PR2 may be transmitted from another device, such as a server that manages the projector 20, via a communication network.

[0121] The acquisition unit 221 acquires, from the control device 10, an image signal corresponding to the projection image PI and a control signal for controlling the projector 20.

[0122] The projection control unit 222 causes the projection device 210 to project a projection image PI corresponding to the image signal acquired by the acquisition unit 221 onto an object based on the control signal acquired by the acquisition unit 221.

[0123] In this embodiment, the projection device 210 is an example of an "optical device." The projector 20 is another example of an "optical device."

[0124] 1-3: Operation of the control device 1-3-1: Overall processing of the control device FIG. 21 is a flowchart showing the overall processing of the control device 10.

[0125] In step S21, the processing device 120 functions as a calibration unit 121. The processing device 120 executes calibration processing between the imaging device 110 and the projector 20. Specifically, the processing device 120 calculates pixel correspondence information indicating the correspondence between the pixels of the projector 20 and the pixels of the imaging device 110, based on the pattern image projected from the projector 20 and the pattern image captured by the imaging device 110. Furthermore, the processing device 120 calculates depth information of the space captured by the imaging device 110, based on the pixel correspondence information.

[0126] In step S22, the processing device 120 functions as a material extraction unit 122. The processing device 120 acquires a first material image MI1 from the material image database MDB. The processing device 120 also removes the image of the background region from the acquired first material image MI1, and generates a second material image MI2 from which only the region to be mapped to the object is extracted.

[0127] In step S23, the processing device 120 functions as the projection target extraction unit 123. The processing device 120 removes the image of the background region from the first object image OI1, and generates a second object image OI2 in which only the region of the object to be projected is extracted.

[0128] In step S24, the processing device 120 functions as a matching processing unit 124. The processing device 120 extracts first feature points CP1 from the second material image MI2. The processing device 120 also extracts second feature points CP2 from the second object image OI2. Furthermore, the processing device 120 generates corresponding point information RI indicating the correspondence between first corresponding points RP1 included in the first feature points CP1 and second corresponding points RP2 included in the second feature points CP2. Details of step S24 will be described later with reference to FIG. 22.

[0129] In step S25, the processing device 120 functions as a geometric correction unit 125. Using the corresponding point information RI, the processing device 120 performs geometric correction on the second material image MI2 so that the coordinates of the first corresponding point RP1 in the second material image MI2 and the coordinates of the second corresponding point RP2 in the second object image OI2 coincide with each other, thereby generating a third material image MI3.

[0130] In step S26, the processing device 120 functions as the projection image generation unit 126. Based on the pixel correspondence information, the processing device 120 converts the coordinate system in the third material image MI3 into a panel coordinate system used in the projector 20, thereby generating a projection image PI having the projection device 210 provided in the projector 20 as the viewpoint.

[0131] In step S27, the processing device 120 functions as the projection control unit 127. The processing device 120 causes the projector 20 to project the projection image PI onto the object.

[0132] 1-3-2: Matching process FIG. 22 is a flowchart showing the matching process in step S24 above.

[0133] In sub-step S241, the processing device 120 acquires a second material image MI2.

[0134] In sub-step S242, the processing device 120 extracts a plurality of first feature points CP1 required for mapping from the second material image MI2. The processing device 120 also generates a first feature map CM1 using the plurality of first feature points CP1.

[0135] In sub-step S243, the processing device 120 obtains a second object image OI2.

[0136] In sub-step S244, the processing device 120 extracts a plurality of second feature points CP2 required for mapping from the second object image OI2, and generates a second feature map CM2 using the plurality of second feature points CP2.

[0137] In sub-step S245, the processing device 120 generates corresponding point information RI indicating the correspondence relationship between corresponding points RP among the multiple first feature points CP1 included in the first feature map CM1 and the multiple second feature points CP2 of the second object image OI2 included in the second feature map CM2.

[0138] 2: Summary of this disclosure A summary of this disclosure is provided below.

[0139] (Supplementary Note 1) A calculation method including: extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, which is input to a first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object, which is input to a second CNN; and deriving correspondences between a plurality of first corresponding points and a plurality of second corresponding points by performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, which are input to a third CNN.

[0140] By separately providing a first CNN for performing a first convolution operation on the second material image MI2 and a second CNN for performing a second convolution operation on the second object image OI2, the first feature points and the second feature points are extracted by different CNNs, thereby improving the accuracy of the correspondence indicated by the corresponding point information RI.

[0141] (Supplementary Note 2) The calculation method according to Supplementary Note 1, wherein the third CNN has a learning model that learns, as training data, correspondences between first corresponding training points included in the plurality of first feature points, which are extracted by inputting a material image at a second resolution obtained by reducing a material image at a first resolution into the first CNN, and second corresponding training points included in the plurality of second feature points, which are extracted by inputting an object image at a second resolution obtained by reducing an object image at the first resolution into the second CNN.

[0142] The third CNN has a learning model trained using, as training data, a dataset consisting of a combination of a material image MI in which the coordinate positions of first feature points are expressed as decimals and an object image in which the coordinate positions of second feature points are expressed as decimals. As a result, when a material image and an object image of a second resolution are input to the learning model, for example, the learning model can output the coordinate positions of the first feature points and the coordinate positions of the second feature points as decimals. In other words, the calculation method of Supplementary Note 2 can generate highly accurate correspondences including decimals.

[0143] (Appendix 3) The calculation method according to appendix 1 or 2, wherein the number of channels of the first CNN decreases continuously or stepwise from the input convolutional layer to the output convolutional layer, and the number of channels of the second CNN decreases continuously or stepwise from the input convolutional layer to the output convolutional layer.

[0144] This allows the third CNN to generate highly accurate correspondences.

[0145] (Supplementary Note 4) A projection method including: extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, the material image being input to a first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object, the object being input to a second CNN; deriving a correspondence relationship between the plurality of first corresponding points and the plurality of second corresponding points by performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, the third CNN; performing geometric correction on the material image using the correspondence relationship; and projecting a projection image in which a coordinate system in the geometrically corrected material image is converted into a panel coordinate system, using an optical device.

[0146] By separately providing a first CNN for performing a first convolution operation on the second material image and a second CNN for performing a second convolution operation on the second object image, the first feature points and the second feature points are extracted by different CNNs, respectively. As a result, the projection system 1 can perform highly accurate matching.

[0147] (Supplementary Note 5) A control device that executes the following operations: extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, which is input to a first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object, which is input to a second CNN; and deriving correspondences between a plurality of first corresponding points and a plurality of second corresponding points by performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, which are input to a third CNN.

[0148] By separately providing a first CNN for performing a first convolution operation on the second material image MI2 and a second CNN for performing a second convolution operation on the second object image OI2, the first feature points and the second feature points are extracted by different CNNs, thereby improving the accuracy of the correspondence indicated by the corresponding point information RI. [Explanation of symbols]

[0149] 1: projection system, 10: control device, 20: projector, 110: imaging device, 120: processing device, 121: calibration unit, 122: material extraction unit, 123: projection target extraction unit, 124: matching processing unit, 124A: matching processing unit, 124B: matching processing unit, 124b[1], 124c[1], 124d[1], 124e[1], 124f[1], 124g[1]: first CNN, 124b[2], 124c[2], 124d[2], 124e[2], 124f[2], 124g[2]: second CNN : 124b[3], 124c[3], 124d[3], 124e[3], 124f[3], 124g[3]: Third CNN, 124c

[11] , 124c

[12] , 124e

[11] , 124e

[12] , 124f

[11] , 124f

[12] : First sub-CNN, 124c

[21] , 124c

[22] , 124e

[21] , 124e

[22] , 124f

[21] , 124f

[22] : Second sub-CNN, 124f: Matching mechanism, 124g: Matching mechanism, 125: Geometric correction unit, 126: Projection image generation unit, 127 : projection control unit, 140: storage device, 150: display device, 160: input device, 170: communication device, 210: projection device, 220: processing device, 221: acquisition unit, 222: projection control unit, 230: storage device, 240: communication device, BI: combined image, CM: feature map, CM1: first feature map, CM2: second feature map, CP: feature point, CP1: first feature point, CP2: second feature point, La1: first layer, La2: second layer, La3: third layer, La4: fourth layer, La5: fifth layer, Lb1: first layer, Lb2: second layer, Lb3: third layer, Lb 4: 4th layer, Lb5: 5th layer, Lc1: 1st layer, Lc2: 2nd layer, Lc3: 3rd layer, Lc4: 4th layer, Lc5: 5th layer, Lc6: 6th layer, MDB: Material image database, MI: Material image, MI1: 1st material image, MI2: 2nd material image, MI3: 3rd material image, OI: Object image, OI1: 1st object image, OI2: 2nd object image, PI: Projected image, PR1: Control program, PR2: Control program, RI: Corresponding point information, RP: Corresponding point, RP1: 1st corresponding point, RP2: 2nd corresponding point, TI: Person image, WI: Clothing image

Claims

1. extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, the material image being input to the first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object input to a second CNN; performing a third convolution operation on a plurality of first corresponding points included in the plurality of first feature points and a plurality of second corresponding points included in the plurality of second feature points, which are input to a third CNN, to derive correspondence relationships between the plurality of first corresponding points and the plurality of second corresponding points; Calculation method including.

2. The third CNN is A first corresponding point for learning included in the plurality of first feature points extracted by inputting a material image of a second resolution obtained by reducing a material image of a first resolution into the first CNN; a learning model that learns, as training data, correspondences between the second feature points and second corresponding points for learning that are extracted by inputting the object image of the first resolution obtained by reducing the object image of the first resolution into the second CNN; The calculation method according to claim 1 .

3. The number of channels of the first CNN decreases continuously or stepwise from the input convolution layer to the output convolution layer, The number of channels of the second CNN decreases continuously or stepwise from the input convolutional layer to the output convolutional layer. The calculation method according to claim 1 or 2.

4. extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, the material image being input to the first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object input to a second CNN; performing a third convolution operation on a plurality of first corresponding points included in a plurality of first feature points and a plurality of second corresponding points included in a plurality of second feature points input to a third CNN, thereby deriving correspondence relationships between the plurality of first corresponding points and the plurality of second corresponding points; performing geometric correction on the material image using the correspondence relationship; a projection image obtained by converting the coordinate system of the geometrically corrected material image into a panel coordinate system, and projecting the projection image using an optical device; A projection method including:

5. extracting a plurality of first feature points by performing a first convolution operation on a material image to be projected onto an object, the material image being input to the first CNN; extracting a plurality of second feature points by performing a second convolution operation on an object image including the object input to a second CNN; performing a third convolution operation on a plurality of first corresponding points included in a plurality of first feature points and a plurality of second corresponding points included in a plurality of second feature points input to a third CNN, thereby deriving correspondence relationships between the plurality of first corresponding points and the plurality of second corresponding points; A control device that executes the above.

Images