Assembly support apparatus and assembly support method

Abstract

To support an assembly operation for a user even when the user cannot obtain assembly product information.SOLUTION: An assembly support apparatus 1 comprises: a first acquisition unit 10 for acquiring an image I1 of the parts of a support target image obtained by picking up plural parts before assembly at an arbitrary arrangement and at arbitrary picking up angles; a generation unit 12 provided with a trained generator G1' constructed by using cycle GAN (Generative Adversarial Networks) and generating an image I2 of a structure in a second level domain from the image I1 of the plural parts in a first domain, and giving the image I1 of the parts of the support target image to the trained generator G1'so as to generate the image I2 of the structure; and a presentation unit 17 for presenting assembly support information including the generated image I2 of the structure.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an assembly assistance device and an assembly assistance method. [Background technology]

[0002] It has become common to purchase products from e-commerce sites. Some products sold on e-commerce sites are delivered to the purchaser in a package with parts and assembly instructions, and the purchaser is provided with a finished product by assembling the parts according to the assembly instructions. For example, furniture such as tables and beds, as well as bicycles, are sometimes provided as assembled products. For users who purchase assembled products, the time and effort required for assembly can be significant. Therefore, Patent Document 1 discloses assembly-ready furniture that reduces the number of components that make up the furniture, thereby reducing the time and effort required for assembly.

[0003] Many assembled products, such as furniture and bicycles, are used over a long period of time, and during that time, the user's living environment may change, and these assembled products may need to be disassembled into parts when moving or renovating their home.When a user attempts to reassemble the product, they may have already lost the assembly instructions, or the model of the assembled product may be outdated, and the electronic data of the assembly instructions or images of the finished product may not be available on the e-commerce site or manufacturer's website from which the product was purchased.As a result, it can be a great deal of effort to reassemble the disassembled parts into a finished product. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5931269 [Non-patent literature]

[0005] [Non-Patent Document 1] Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”, ICCV, pages 2223-2232, 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] Thus, in the prior art, assembly work was labor intensive, especially when assembly information was no longer available to the user.

[0007] The present invention has been made to solve the above-mentioned problems, and has an object to support the user's assembly work even when assembly information is not available to the user. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the assembly support device of the present invention is an assembly support device that supports the assembly of a structure formed by assembling a plurality of parts, and includes: a first acquisition unit configured to acquire an support target image obtained by photographing the plurality of parts before assembly at an arbitrary arrangement and shooting angle; a trained generator constructed using cycle GAN (Generative Adversarial Networks) that generates an image of a structure in a second domain from images of the plurality of parts in a first domain, a generation unit configured to provide the support target image to the trained generator to generate an image of the structure; and a presentation unit configured to present assembly support information including the generated image of the structure.

[0009] In addition, the assembly assistance device according to the present invention may further include a learning unit configured to use the cycle GAN to train a generator that generates images of the structure in the second domain from images of the plurality of parts in the first domain, and the learning unit may use a plurality of images of the plurality of parts photographed in different arrangements and at different photographing angles as training data for images of the plurality of parts in the first domain.

[0010] Moreover, the assembly assistance device according to the present invention may further include an identification unit configured to identify identification information that identifies the structure, which is included in the image of the structure generated by the generation unit, and a second acquisition unit configured to acquire, based on the identified identification information, an assembly procedure for the structure from a storage unit in which identification information and assembly procedures are stored in association with each other, and the presentation unit may present the assembly assistance information that includes the acquired assembly procedure for the structure and an image of the structure.

[0011] In order to solve the above-mentioned problems, the assembly assistance method of the present invention is an assembly assistance method for assisting in the assembly of a structure formed by assembling a plurality of parts, and includes a first acquisition step of acquiring an assistance target image in which the plurality of parts before assembly are photographed at an arbitrary arrangement and shooting angle; a generation step of providing the assistance target image to a trained generator constructed using cycle GAN (Generative Adversarial Networks) that generates an image of a structure in a second domain from images of a plurality of parts in a first domain, to generate an image of the structure; and a presentation step of presenting assembly assistance information including the generated image of the structure.

[0012] In addition, the assembly assistance method according to the present invention may further include a learning step of using the cycle GAN to train a generator that generates images of the structure in the second domain from images of the plurality of parts in the first domain, and the learning step may use a plurality of images of the plurality of parts photographed in different arrangements and at different photographing angles as training data for images of the plurality of parts in the first domain.

[0013] Furthermore, the assembly assistance method according to the present invention may further include a specifying step of specifying identification information that identifies the structure, which is included in the image of the structure generated in the generating step, and a second obtaining step of obtaining an assembly procedure for the structure from a storage unit in which identification information and assembly procedures are stored in association with each other, based on the specified identification information, and the presenting step may present the assembly assistance information including the obtained assembly procedure for the structure and an image of the structure. [Effects of the Invention]

[0014] According to the present invention, a trained generator, which is constructed using cycle GAN and generates an image of a structure in a second domain from images of multiple parts in a first domain, is provided with an image of a support target to generate an image of the structure. Therefore, even if the information of the assembly product is not available to the user, the user can be supported in the assembly work. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing the configuration of an assembly assist system including an assembly assist device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the configuration of a learning unit included in the assembly assisting device according to this embodiment. [Figure 3] FIG. 3 is a diagram for explaining the configuration of the learning unit included in the assembly assisting device according to this embodiment. [Figure 4] FIG. 4 is a diagram for explaining the configuration of the learning unit included in the assembly assisting device according to this embodiment. [Figure 5] FIG. 5 is a block diagram showing the hardware configuration of the assembly assisting device according to this embodiment. [Figure 6] FIG. 6 is a sequence diagram showing an outline of the operation of the assembly assist system according to this embodiment. [Figure 7] FIG. 7 is a flowchart showing the learning process performed by the assembly assisting device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to FIGS.

[0017] [Configuration of assembly support system] First, an overview of an assembly assistance system including an assembly assistance device 1 according to an embodiment of the present invention will be described. In the following, a structure refers to an assembly in which a plurality of parts are assembled into a finished product. In the following, it is assumed that an image of the finished structure is not available to the user.

[0018] 1 is a block diagram showing the configuration of an assembly support system including an assembly support device 1 according to an embodiment of the present invention. The assembly support system according to this embodiment includes the assembly support device 1, a user terminal 2, and a server 3. The assembly support device 1, the server 3, and the user terminal 2 are connected to each other so as to be able to communicate with each other via a network NW such as a LAN or a WAN. The assembly support system supports the assembly of a structure formed by assembling a plurality of parts.

[0019] [User device configuration] The user terminal 2 can be realized by a computer including a processor 20, a main memory device 21, a communication interface 22, and an auxiliary memory device 23, and a program that controls these hardware resources. The user terminal 2 also includes a display device 24 and a camera 25. The user terminal 2 can further include a speaker (not shown).

[0020] The user terminal 2 can be realized as a mobile communication terminal such as a smartphone, a tablet computer, a laptop computer, or the like. A user requesting assembly assistance uses the camera 25 of the user terminal 2 to capture an image I1 of parts, in which multiple parts are arbitrarily arranged and photographed at an arbitrary angle, as an assistance target image. The image I1 of parts is an image in which all parts included in an assembly before assembly are arbitrarily arranged by the user and photographed at an arbitrary photographing angle. In the example of the image I1 of parts shown in FIG. 1, the tabletop and multiple legs that make up a table are arranged separately and photographed at a high angle by the camera 25 of the user terminal 2.

[0021] The display device 24 has a liquid crystal display or an organic EL display. The display device 24 can also function as a touch panel that accepts input operations from the user. The display device 24 displays an image I1 of a part photographed by the user using the camera 25, and also displays an image I2 of a structure in which multiple parts are assembled, received from the assembly support device 1.

[0022] Camera 25 is a digital camera that can capture an image I1 of a component to be supported in assembly as a color image in which the color of each pixel is expressed in RGB. Camera 25 has a certain number of pixels and resolution, such as 10 million pixels. In this embodiment, image I1 of the component captured by camera 25 is subjected to image processing by processor 20, for example, by removing the background or making it white, so that the image contains no images other than the component.

[0023] [Server Configuration] The server 3 includes a third storage unit 30. The third storage unit 30 stores previously collected identification information for identifying the type of assembly product, such as the product number or model number of the assembly product for which assembly support is being provided, in association with data on the assembly instruction manual. In addition to the assembly instruction manual data, the third storage unit 30 can also store audio guidance in which the assembly procedures included in the assembly instruction manual are converted into audio. In response to a request from the assembly support device 1, the server 3 sends to the assembly support device 1 the data on the assembly instruction manual for the assembly product for which assembly support is being provided, the product number or model number.

[0024] [Function block of assembly support device] 1, the assembly support device 1 includes a first acquisition unit 10, a learning unit 11, a generation unit 12, a first storage unit 13, an identification unit 14, a second storage unit 15, a second acquisition unit 16, and a presentation unit 17. The assembly support device 1 generates an image I2 of a structure in which the parts are assembled from an image I1 of the parts transmitted from the user terminal 2 using a trained generator G1' constructed in advance by cycle GAN, and presents assembly support information including the image I2 of the structure.

[0025] The first acquisition unit 10 acquires, as an assistance target image, an image I1 of a plurality of parts before assembly, photographed at an arbitrary arrangement and angle. The first acquisition unit 10 acquires the image I1 of the parts transmitted from the user terminal 2 via the network NW. The first acquisition unit 10 performs preprocessing on the acquired image I1 of the parts, such as removing noise and converting the size and resolution.

[0026] The learning unit 11 performs learning of a cycle GAN having two generators G1 and G2 and two classifiers D1 and D2, as shown in Fig. 2. The cycle GAN is a type of GAN (Generative Adversarial Network), and is a generative model that can perform image conversion between different domains (see Non-Patent Document 1).

[0027] While GAN is a model with one generator and one classifier, Cycle GAN is composed of two generators G1 and G2 and two classifiers D1 and D2, with one generator and one classifier for each domain. That is, for the first domain, the generator G1 and the classifier D2 are provided, and for the second domain, the generator G2 and the classifier D1 are provided.

[0028] The learning unit 11 uses cycle GAN to train a generator G1 that generates an image I2 of a structure in a second domain from an image I1 of a part in a first domain. A domain refers to a collection of data with certain characteristics. The image I1 of the part in a disassembled state of the structure and the image I2 of the structure in a completed state have different characteristics and are images in different domains with no pixel-by-pixel correspondence.

[0029] The cycle GAN includes a pair of a generator G1 that converts an image I1 of a part in a first domain into an image I2 of a structure in a second domain, and a classifier D1 that determines the authenticity of the image I2 of the structure converted by the generator G1, and a pair of a generator G2 that inversely converts the image I2 of the structure in the second domain into an image I1 of a part in the first domain, and a classifier D2 that determines the authenticity of the inversely converted image I1 of the part. In this way, the cycle GAN has a cyclic structure of conversion and inverse conversion using two pairs of generators G1 and G2 and classifiers D1 and D2 to learn the relationship between the first domain and the second domain.

[0030] 3 and 4 are diagrams showing the neural network configurations of generators G1 and G2 and discriminators D1 and D2 of the cyclic GAN used by the learning unit 11. The generators G1 and G2 are configured as neural networks having an input layer, a hidden layer, and an output layer. CNN or ResNet can be used as the neural networks configuring the generators G1 and G2.

[0031] FIG. 3 is a diagram showing a schematic representation of the neural network configuration of the generator G1. As shown in FIG. 3, the values ​​of each pixel constituting the image I1 of the part of the first domain are input to the input node of the generator G1. The values ​​of each pixel (x1 to x n ) is input. The generator G1 performs a product-sum operation on the input and weight parameters and threshold processing using an activation function to generate output data G1(x1) to G1(x n ) The output data from generator G1 is image I2 of the structure in the second domain, converted from image I1 of the part in the first domain. Similarly, generator G2 receives the values ​​of each pixel that make up image I2 of the structure in the second domain, and then performs product-sum operations on the input and weight parameters and threshold processing using an activation function to obtain output data for the first domain.

[0032] Similarly, the classifiers D1 and D2 are also configured with neural networks having an input layer, a hidden layer, and an output layer. A CNN can be used as the neural network that configures the classifiers D1 and D2. Figure 4 is a diagram that schematically illustrates the neural network configuration of the classifier D1. The input node of the classifier D1 is given the values ​​of each pixel that configures the image I2 of the structure. In the example of Figure 4, the output data G1(x1) to G1(x n ) is given to the input node of the discriminator D1. The output data G1(x1) to G1(x n ) are the values ​​of each pixel that make up the image I2 of the structure generated by the generator G1.

[0033] Classifier D1 performs a product-sum operation on the input and weight parameters and threshold processing using an activation function, and then outputs a binary output of 1 or 0. An output value of 1 indicates that the structure image I2 is a genuine image, while an output value of 0 indicates that the input structure image I2 is a pseudo-image.

[0034] Similarly, for the first domain classifier D2, the value of each pixel constituting the first domain part image I1 is input to the input node, and the neural network outputs a value of 1 or 0 as the calculation result. An output value of 1 indicates that the input part image I1 is a genuine image, and an output value of 0 indicates that the input part image I1 is a pseudo image.

[0035] Here, the learning unit 11 introduces an objective function that combines an adversarial loss and a cycle consistency loss to learn the relationship between the first domain and the second domain using cycle GAN. Similar to a conventional GAN, an adversarial loss is used in learning the transformation of an image from the first domain to the second domain and the inverse transformation of an image from the second domain to the first domain. In addition to this adversarial loss, the learning unit 11 incorporates a cycle consistency loss into the objective function to learn a relationship in which a generator G1 that performs image transformation from the first domain to the second domain and a generator G2 that performs inverse transformation from the second domain to the first domain maintain consistency in the cycle of repeated transformation and inverse transformation.

[0036] The adversarial loss used by the learning unit 11 will be explained below by taking as an example a network made up of a generator G2 that converts images from the second domain to the first domain and a classifier D2 that judges the authenticity of the converted image. First, let us denote the data of the real image of the part image I1 in the first domain as x, the predicted value output by the classifier D2 as y, and the correct label as t. The correct label t is set to 1 for the real image of the part image I1 in the first domain, and 0 for the pseudo image of the part image I1 generated by the generator G2. In this case, the classifier D2 calculates the cross entropy E in the following equation (1) for a binary classification problem: CE It can be expressed as:

[0037]

number

[0038] The first term in the brace of the above equation (1) represents t n lny nIn this case, the predicted value y n However, the correct label t of the real image of the part image I1 is n = 1. On the other hand, the second term in the braces represents (1-t n )ln(1-y n ), the predicted value y n is the correct label value (1-t n ) = 0. In this way, the cross entropy E CE is the maximum value when the predicted value matches the correct label value.

[0039] Here, the generators G1 and G2 (hereinafter collectively referred to as “generator G”) that make up the cycle GAN are G ,θ G and the function G(w G ,θ G ) and the classifiers D1 and D2 (hereinafter collectively referred to as "classifier D") are expressed as follows: D ,θ D and function D(w D ,θ D ) The cross entropy E in the above equation (1) CE The adversarial loss L of the model with a generator G and a discriminator D based on GAN can be expressed by the following equation (2).

number

[0040] The first term of the above equation (2) represents E D(x)=1 lnD(w D ,θ D ) is the expected value that the classifier D will classify a real image as real. D(x)=0 ln(1-D(G(w G ,θ G ),w D ,θ D)) is the expected value that the classifier D will classify the pseudo-image generated by the generator G as a pseudo-image. In the adversarial learning of cycle GAN, the adversarial loss L for each pair of the generator G and the classifier D is GAN The generator G and the classifier D are adversarially trained using min-max optimization. Therefore, the generator G is trained to generate pseudo-images that can fool the classifier D, and the classifier D is trained to identify the pseudo-images generated by the generator G as pseudo-images.

[0041] In the adversarial learning of the classifier D, when a real image is given, the classifier D outputs an output close to y=1, and the adversarial loss L in the above equation (2) is GAN On the other hand, when a pseudo-image is given, the adversarial loss L GAN Learning is performed to maximize the second term of

[0042] In the adversarial learning of the generator G, D(G(w G ,θ G ),w D ,θ D ) is close to 1. G ,θ G ) to obtain the adversarial loss L GAN The learning unit 11 uses a learning procedure in which the parameters of the generator G and the parameters of the discriminator D are updated alternately.

[0043] Here, as shown in Figure 2, the generator G1(w G1 ,θ G1 ) is input with the real image x of the part image I1, and the converted output is G1(x), and the generator G2(w G2 ,θ G2 ) is input with the real image y of the structure image I2, and the converted output is represented as G2(y). D1 ,θ D1 ) when a genuine image y of the structure image I2 is input, the output of the authenticity judgment is represented as D1(y).

[0044] Furthermore, the discriminator D1(w D1 ,θD1 ), generator G1(w G1 ,θ G1 ) is input, the output of the true / false judgment when the output G1(x) which is a pseudo image of the image I2 of the structure generated by the classifier D2(w D2 ,θ D2 ) is input with the real image x of the part image I1, the output of the authenticity judgment is D2(x), and the classifier D2(w D2 ,θ D2 ), generator G2(w G2 ,θ G2 ) is input, the output of the true / false judgment when the output G2(y) which is a pseudo image of the image I1 of the part generated in step (1) is input is represented as D2(G2(y)).

[0045] From these definitions, the adversarial loss L in Equation (2) above is GAN is the adversarial loss L in the following equation (3). GAN1 , and the adversarial loss L in the following equation (4) GAN2 It is expressed as:

[0046]

number

[0047] Using the above equation (3), the image I1 of the part in the first domain is converted into the image I2 of the structure in the second domain by the generator G1, and the adversarial loss L of the network that judges the authenticity of the converted structure image I2 by the discriminator D1 is GAN1 Furthermore, by using the above formula (4), the image I2 of the structure in the second domain is converted into the image I1 of the part in the first domain by the generator G2, and the adversarial loss L of the network that judges the authenticity of the converted part image I1 by the classifier D2 is calculated. GAN2 is calculated.

[0048] The learning unit 11 calculates the adversarial loss L of the network related to the generator G1 and the discriminator D1 in the above formula (3). GAN1 , and the adversarial loss L of the network related to the generator G2 and the discriminator D2 in the above equation (4). GAN2 Optimize each of them.

[0049] The learning unit 11 further calculates the cycle consistency loss L cyc The input image is transformed by generator G using the generator G, and the transformed image is then inversely transformed by another generator G to reconstruct the original image, which is then trained to match the input image before transformation.

[0050]

number

[0051] The cycle consistency loss L in equation (5) above cyc is the norm L 1 The loss is expressed in the form of an expectation using the formula (5), and is calculated by comparing the image restored by inversely transforming an image transformed by generator G with the original input image, pixel by pixel. The first term in the formula (5) evaluates whether the original image x of the part image I1 is input as an image and transformed by generators G1 and G2 in that order, and whether it returns to the part image I1, based on the output G2(G1(x)) after transformation and the absolute value of x. The loss in this first term is called the cycle consistency loss L cyc1 The second term uses the real image y of the structure image I2 as the input image and transforms it in the order of generator G2 and G1, and evaluates whether it returns to the structure image I2 based on the output G1(G2(y)) after transformation and the absolute value of y. The loss of the second term is the cycle consistency loss L cyc2 That's what they say.

[0052] The learning unit 11 calculates the adversarial loss L GAN1 , L GAN2 and the cycle consistency loss L in equation (5) above. cyc By combining these, we set the final objective function L of the cycle GAN, which is expressed as the following equation (6).

number

[0053] Returning to Figure 1, the generation unit 12 includes a trained generator G1' constructed using cycle GAN to generate an image I2 of a structure in a second domain from an image I1 of multiple parts in a first domain, and the trained generator G1' is given an image I1 of a part, which is an image to be supported, to generate an image I2 of the structure.

[0054] The first storage unit 13 stores the trained generator G1′. The first storage unit 13 can also store the pre-training generators G1 and G2 and the discriminators D1 and D2 that constitute the cycle GAN.

[0055] The identification unit 14 identifies identification information for identifying a structure included in the image I2 of the structure generated by the generation unit 12. More specifically, the identification unit 14 can extract character strings such as the model number of an assembly included in the image I1 of the part or the image I2 of the structure acquired by the first acquisition unit 10, and convert them into character data. Furthermore, the identification unit 14 can identify identification information such as the model number, product number, and manufacturer of the assembly related to the part and the structure based on the character data, with reference to the model number information stored in the second storage unit 15.

[0056] Alternatively, the identification unit 14 can classify identification information including the model number of the assembly by image classification or object detection based on the image I2 of the structure generated by the generation unit 12. In this case, the identification unit 14 can train a machine learning model in advance using the image I2 of the structure used as training data by the learning unit 11 and teacher data in which the identification information including the model number of the assembly is used as a correct answer label. The identification unit 14 can identify identification information from the image I2 of the structure using the trained machine learning model.

[0057] The second storage unit 15 stores identification information of assemblies corresponding to parts and structures, such as model numbers and manufacturers of the assemblies.

[0058] The second acquisition unit 16 acquires the assembly procedures for the structure from the third storage unit 30 (storage unit) in which the identification information and the assembly procedures are stored in association with each other, based on the identification information of the structure identified by the identification unit 14. More specifically, the second acquisition unit 16 acquires data of the assembly instructions for the structure from the external server 3 via the network NW, based on the identification information including the model number of the assembly product related to the structure. The information related to the assembly procedures acquired by the second acquisition unit 16 includes text information, audio, etc.

[0059] The presentation unit 17 presents assembly support information including an image I2 of the structure generated by the generation unit 12. Furthermore, the presentation unit 17 can present assembly support information including an assembly procedure for the structure acquired by the second acquisition unit 16 and the image I2 of the structure. The presentation unit 17 transmits the assembly support information to the user terminal 2 via the network NW.

[0060] [Hardware configuration of assembly support device] Next, an example of a hardware configuration for realizing the assembly assisting device 1 having the above-described functions will be described with reference to FIG.

[0061] 5, the assembly support device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output (I / O) 106, which are connected via a bus 101, and a program that controls these hardware resources. The assembly support device 1 can also include a display device 107 connected via the bus 101.

[0062] The main memory device 103 pre-stores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 implement the functions of the assembly assist device 1, such as the first acquisition unit 10, the learning unit 11, the generation unit 12, the identification unit 14, the second acquisition unit 16, and the presentation unit 17 shown in FIG.

[0063] The communication interface 104 is an interface circuit for connecting the assembly assisting device 1 to various external electronic devices via a network.

[0064] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or flash memory as the storage medium.

[0065] The auxiliary storage device 105 has a program storage area for storing the cycle GAN learning program and assembly assistance program executed by the assembly assistance device 1. The auxiliary storage device 105 realizes the first storage unit 13 and the second storage unit 15 described in FIG. 1. Furthermore, for example, the auxiliary storage device 105 may have a backup area for backing up the above-mentioned data, programs, etc.

[0066] The input / output I / O 106 is an input / output device that inputs signals from external devices and outputs signals to external devices.

[0067] The display device 107 is configured by an organic EL display, a liquid crystal display, etc. The display device 107 can also realize the presentation unit 17.

[0068] [Assembly support device operation] Next, the operation of the assembly assisting device 1 having the above-described configuration will be described with reference to the sequence diagram of FIG. 6 and the flowchart of FIG.

[0069] 6 is an operation sequence showing an outline of the operation of an assembly support system including the assembly support device 1. First, the learning unit 11 of the assembly support device 1 uses cycle GAN to learn a generator G1 that generates an image I2 of a structure in a second domain from an image I1 of a plurality of parts in a first domain (step S1). Details of the learning process in step S1 will be described later.

[0070] Thereafter, the user requesting assembly assistance uses the camera 25 of the user terminal 2 to photograph the multiple parts of the disassembled assembly at an arbitrary arrangement and angle (step S2). The user terminal 2 transmits the photographed image I1 of the parts as an assistance target image to the assembly assistance device 1 via the network NW.

[0071] In step S2, the user terminal 2 can transmit information about the assembly for which assembly assistance is requested, along with the image I1 of the parts. For example, the user terminal 2 can accept an input operation of information such as the manufacturer of the assembly and the year of purchase, and transmit this information to the assembly assistance device 1 along with the image I1 of the parts. In addition, the user terminal 2 can transmit attributes of the assembly for which assistance is required, along with the image I1 of the parts. The attributes of the assembly include categories such as furniture and bicycles, and more detailed categories of furniture such as shelves and tables. The information about the assembly is used in step S5, which will be described later.

[0072] Next, the first acquisition unit 10 of the assembly support device 1 acquires the part image I1 from the user terminal 2 (step S3). In step S3, the first acquisition unit 10 performs preprocessing on the acquired part image I1, such as removing noise and converting the size and resolution.

[0073] Next, the generator 12 of the assembly support device 1 provides the image I1 of the part, which is the support target image, to the trained generator G1′ constructed in advance in the learning process of step S1 to generate an image I2 of the structure (step S4).

[0074] Next, the identification unit 14 identifies identification information for identifying the structure included in the image I2 of the structure generated in step S3 (step S5). The identification unit 14 extracts character strings such as the model number of the assembly included in the image I1 of the part or the image I2 of the structure acquired in step S3, converts them into character data, and can identify identification information such as the model number and manufacturer of the assembly related to the part and the structure by referring to the model number information stored in the second storage unit 15. The identification unit 14 can also identify the identification information by using the information related to the assembly transmitted in step S2.

[0075] Next, the second acquisition unit 16 acquires the assembly procedure of the structure from the server 3 including the third storage unit 30 that stores the identification information and the assembly procedure in association with each other, based on the identification information of the structure identified in step S5 (step S6). The information related to the assembly procedure acquired in step S6 includes text information, audio, etc.

[0076] Next, the presentation unit 17 presents assembly support information including the assembly procedure of the structure acquired in step S6 and the image I2 of the structure generated in step S4 to the user terminal 2 (step S7). After that, the user terminal 2 that has received the assembly support information causes the display device 24 to display the image I2 of the structure and the assembly procedure (step S8). The user who requested assembly support can refer to the image I2 of the structure and the assembly procedure displayed on the screen of the user terminal 2 and assemble the completed structure from the parts.

[0077] Next, the learning process performed by the assembly assisting device 1 will be described with reference to the flowchart of FIG. 7 and the block diagram of the learning unit 11 of FIG.

[0078] First, the learning unit 11 prepares training data (step S100). real , and the real image y of the structure image I2 realare used as training data. In this embodiment, a plurality of images of a plurality of parts photographed at different positions and different photographing angles are used as training data. Also, a plurality of images of a structure photographed at different photographing angles are used as training data. These training data are set as "input x" and "input y" in FIG. 2, respectively.

[0079] Next, the learning unit 11 inputs training data to two classifiers D1 and D2 constituting the cycle GAN, respectively, to have them learn real images, and calculates the parameter w D1 ,θ D1 , and the parameter w of the discriminator D2 D2 ,θ D2 In step S101, a plurality of real images y of the structure taken at different angles are updated. real is input to the classifier D1, and the parameter w is calculated using the backpropagation method or the like so that the tolerance entropy error in the above formula (1) is minimized. D1 ,θ D1 Similarly, for the classifier D2, multiple real images x of multiple parts taken at different positions and angles are updated. real is given as input, and the parameter w is calculated using backpropagation or other methods to minimize the tolerance entropy error in equation (1) above. D2 ,θ D2 Update.

[0080] Next, the learning unit 11 performs adversarial learning on a network consisting of a pair of the generator G1 and the classifier D1, and on a network consisting of a pair of the generator G2 and the classifier D2, in the following steps.

[0081] First, the learning unit 11 calculates the adversarial loss L GAN1 In more detail, the learning unit 11 trains the classifier D1 using the real image y of the structure image I2 as shown in FIG. real is given as input, and the adversarial loss L GAN1 The gradient dL GAN1 / dw D1 ,dL GAN1 / dθD1 , and calculate the adversarial loss L GAN1 parameter w D1 ,θ D1 Furthermore, the learning unit 11 updates the pseudo-image y of the image I2 of the structure generated by the generator G1 in the classifier D1. fake is given as input, and similarly the adversarial loss L GAN1 parameter w D1 ,θ D1 Note that the learning of the classifier D1 is performed by updating the parameter w G1 ,θ G1 is fixed.

[0082] Next, the learning unit 11 calculates the adversarial loss L GAN1 In more detail, the learning unit 11 trains the generator G1 using the real image x of the part image I1 as shown in FIG. real is given as input, and the adversarial loss L GAN1 Gradient of -dL GAN1 / dw G1 ,-dL GAN1 / dθ G1 Calculate the adversarial loss L by backpropagation or other methods. GAN1 The parameter w is set to be minimum. G1 ,θ G1 The learning of the generator G1 is performed by updating the parameter w D1 ,θ D1 is fixed.

[0083] Next, the learning unit 11 calculates the adversarial loss L GAN2 In more detail, as shown in FIG. 2, the learning unit 11 trains the classifier D2 using the real image x of the part image I1. real is given as input, and the adversarial loss L GAN2 The gradient dL GAN2 / dw D2 ,dL GAN2 / dθ D2 , and calculate the adversarial loss L GAN2parameter w D2 ,θ D2 Furthermore, the learning unit 11 updates the pseudo-image y of the part image I1 generated by the generator G2 in the classifier D2. fake is given as input, and similarly the adversarial loss L GAN2 parameter w D2 ,θ D2 Note that the learning of the classifier D2 is performed by updating the parameter w G2 ,θ G2 is fixed.

[0084] Next, the learning unit 11 calculates the adversarial loss L GAN2 In more detail, the learning unit 11 trains the generator G2 using the real image y of the structure image I2 as shown in FIG. real is given as input, and the adversarial loss L GAN2 Gradient of -dL GAN2 / dw G2 ,-dL GAN2 / dθ G2 , and calculate the adversarial loss L GAN2 The parameter w is set to be minimum. G2 ,θ G2 The learning of the generator G2 is performed by updating the parameter w D2 ,θ D2 The learning unit 11 performs the learning in steps S104 and S105 to obtain the adversarial loss L GAN2 Ask for.

[0085] Next, the learning unit 11 calculates the cycle consistency loss L cyc is calculated in the following steps. First, the learning unit 11 calculates the real image x of the part image I1. real and the cycle consistency loss L between the pseudo-image G2(G1(x)) of the reconstructed part image I1. cyc1 is calculated and the generators G1 and G2 are trained (step S106).

[0086] More specifically, as shown in FIG. 2, the learning unit 11 learns the real image x of the part image I1.real is given to the generator G1, and the generated pseudo image G1(x) of the image I2 of the structure is further given as input to the generator G2. The pseudo image G2(G1(x)) of the image I1 of the part further reconstructed by the generator G2 and the real image x of the image I1 of the part are given as input to the generator G2. real Cycle consistency loss with L cyc1 is calculated by the above formula (5). The learning unit 11 calculates the original image, the real image x real and the reconstructed pseudo image G2(G1(x)) are compared pixel by pixel, and the distance L 1 By minimizing the cycle consistency loss L cyc1 Minimize.

[0087] Next, the learning unit 11 calculates the real image y real and the cycle consistency loss L between the pseudo-image G1(G2(y)) of the reconstructed structure image I2. cyc2 , and trains the generators G1 and G2 (step S107). More specifically, as shown in FIG. 2, the training unit 11 calculates the real image y real is given to the generator G2, and the generated pseudo image G2(y) of the image I1 of the part is further given as input to the generator G1. The pseudo image G1(G2(y)) of the image I2 of the structure further reconstructed by the generator G1 and the real image y of the image I2 of the structure are given as input to the generator G1. real Cycle consistency loss with L cyc2 The learning unit 11 calculates the original image, the real image y real and the reconstructed pseudo image G1(G2(y)) are compared pixel by pixel, and the distance L 1 By minimizing the cycle consistency loss L cyc2 Minimize.

[0088] Next, the learning unit 11 calculates the adversarial loss L obtained in steps S102 to S107. GAN1 , L GAN2 , and cycle consistency loss L cycThen, the objective function L of the above equation (6) is set, and the processes from step S102 to step S107 are repeated until the objective function L converges (step S108: NO). The learning unit 11 stores the generator G1 obtained when the objective function L converges as a trained generator G1' in the first storage unit 13 (step S109). Through the above learning process, a trained generator G1' that converts an image I1 of a part in the first domain into an image I2 of a structure in the second domain is constructed.

[0089] As described above, according to the assembly assistance device 1 of this embodiment, a generator G1 is constructed using cycle GAN to convert an image I1 of a part in a first domain into an image I2 of a structure in a second domain. Therefore, even if information about the assembly product, particularly the image I2 of the structure, is not available to the user, the assembly work of the user can be assisted.

[0090] Furthermore, according to the assembly support device 1 of this embodiment, in addition to the image I2 of the structure generated by the trained generator G1', information on the assembly procedure of the assembly product related to the structure is presented to the user as assembly support information. Therefore, the user can perform the assembly work by referring to the image I2 of the completed structure and the assembly procedure, thereby reducing the labor involved in the user's assembly work.

[0091] Furthermore, according to the assembly support device 1 of this embodiment, multiple images of multiple parts in different arrangements and taken at different shooting angles are used as training data for images I1 of multiple parts in the first domain, so that the user can arrange multiple parts as desired and generate image I2 of a structure from images I1 of parts taken at any shooting angle.

[0092] In the above-described embodiment, the learning unit 11 uses the adversarial loss L GAN , and cycle consistency loss L cyc However, the learning unit 11 sets the objective function L by combining the adversarial loss L GAN , and cycle consistency loss L cycIn addition, an objective function L having a identity loss may be set. Identity loss is, for example, calculating a loss for each pixel so that the input image to the generator G matches the pseudo-image generated by the generator G. By further introducing identity loss, the generator G1 is trained to convert only the necessary parts of the input part image I1.

[0093] In the above-described embodiment, the assembly support device 1 has been described as acquiring the image I1 of the part directly from the user terminal 2 via the network NW. However, the assembly support device 1 may be configured to acquire the image I1 of the part of the support target image from an external device. In this case, the user terminal 2 transmits an assembly support request to the external device together with the image I1 of the part. For example, the server 3 may be configured to accept the assembly support request from the user terminal 2.

[0094] In the above-described embodiment, the assembly support device 1 directly presents assembly support information to the user terminal 2 via the network NW. However, the assembly support device 1 may be configured to present the assembly support information to the user terminal 2 via an external device. For example, the assembly support information may be transmitted to the user terminal 2 via the server 3.

[0095] In the above-described embodiment, the assembly assisting device 1 and the server 3 are described as being independent from each other. However, the assembly assisting device 1 may be configured to include the third storage unit 30 included in the server 3.

[0096] In the above-described embodiment, for ease of explanation, the number of assemblies for which the assembly assistance device 1 provides assembly assistance has been described as one. However, the number of assemblies may be multiple. Therefore, the learning unit 11 can learn a cycle GAN for each assembly to construct a trained generator G1'. In this case, the generation unit 12 can be configured to use an image I2 of a structure that has been accurately generated from among the images generated by providing an image I1 of a part, which is an assistance target image transmitted from the user terminal 2, to multiple generators G1'.

[0097] The above describes embodiments of the assembly assistance device and assembly assistance method of the present invention, but the present invention is not limited to the described embodiments, and various modifications that can be imagined by those skilled in the art can be made within the scope of the invention described in the claims. [Explanation of symbols]

[0098] 1...assembly assistance device, 10...first acquisition unit, 11...learning unit, 12...generation unit, 13...first memory unit, 14...identification unit, 15...second memory unit, 16...second acquisition unit, 17...presentation unit, 2...user terminal, 25...camera, 3...server, 30...third memory unit, 101...bus, 20, 102...processor, 21, 103...main memory unit, 22, 104...communication interface, 23, 105...auxiliary memory unit, 106...input / output I / O, 24, 107...display device, G, G1, G2...generator, D, D1, D2...classifier, G1'...trained generator, NW...network.

Claims

1. An assembly support device that supports the assembly of a structure formed by assembling a plurality of parts, a first acquisition unit configured to acquire an assistance target image obtained by photographing the plurality of components before assembly at an arbitrary arrangement and at an arbitrary photographing angle; a generation unit including a trained generator constructed using cycle Generative Adversarial Networks (GAN) that generates an image of a structure in a second domain from images of a plurality of parts in a first domain, the trained generator being configured to provide the support target image to the trained generator to generate an image of the structure; a presentation unit configured to present assembly support information including the generated image of the structure; An assembly support device comprising:

2. 2. The assembly assist device according to claim 1, a learning unit configured to use the cycle GAN to learn a generator that generates an image of the structure in the second domain from images of the plurality of parts in the first domain; The learning unit uses a plurality of images of the plurality of components, in which the plurality of components are photographed in different positions and at different photographing angles, as training data of images of the plurality of components in the first domain. An assembly support device characterized by:

3. 3. The assembly assisting device according to claim 1, an identification unit configured to identify identification information that identifies the structure included in the image of the structure generated by the generation unit; a second acquisition unit configured to acquire, based on the identified identification information, an assembly procedure for the structure from a storage unit in which identification information and assembly procedures are stored in association with each other; Equipped with The presentation unit presents the assembly support information including the acquired assembly procedure of the structure and an image of the structure. An assembly support device characterized by:

4. An assembly support method for supporting the assembly of a structure formed by assembling a plurality of parts, comprising: a first acquisition step of acquiring an assistance target image obtained by photographing the plurality of components before assembly at an arbitrary arrangement and at an arbitrary photographing angle; a generation step of providing the support target image to a trained generator that generates an image of a structure in a second domain from images of a plurality of parts in a first domain, the trained generator being constructed using cycle GAN (Generative Adversarial Networks), and causing the trained generator to generate an image of the structure; a presentation step of presenting assembly support information including the generated image of the structure; An assembly assistance method comprising:

5. 5. The assembly support method according to claim 4, further comprising a learning step of using the cycle GAN to learn a generator that generates an image of the structure in the second domain from images of the plurality of parts in the first domain; The learning step uses a plurality of images of the plurality of components taken at different positions and different angles as training data of images of the plurality of components in the first domain. An assembly assistance method comprising:

6. 6. The assembly support method according to claim 4, further, a specifying step of specifying identification information for identifying the structure included in the image of the structure generated in the generating step; a second acquisition step of acquiring an assembly procedure for the structure from a storage unit in which identification information and assembly procedures are stored in association with each other based on the identified identification information; Equipped with The presenting step presents the assembly support information including the acquired assembly procedure of the structure and an image of the structure. An assembly assistance method comprising:

Images