Information output system and control program for information output system

The information output system addresses the inconvenience of ordering building materials by using image-based machine learning to identify and specify products, enhancing accuracy and convenience in material selection.

JP7594066B1Active Publication Date: 2024-12-03KOMATSU CORP +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023150356
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-09-15
Publication Date
2024-12-03
Estimated Expiration
2043-09-15

AI Technical Summary

Technical Problem

The conventional method of ordering building materials is inconvenient due to the large number of products in catalogs, discrepancies between catalog photos and actual products, and the difficulty in identifying and finding specific materials, especially when they are not listed in the latest catalog version.

Method used

An information output system that utilizes an image acquisition unit to capture images of building materials, applies a trained model generated by machine learning to identify the material, and outputs information such as product details and purchase URLs, incorporating supervised learning with varied illumination and image cropping techniques to enhance accuracy.

Benefits of technology

Facilitates easy identification of desired building materials by eliminating the need to visually search through numerous products and ensures accurate recognition even when the material is not listed in the latest catalog, thereby improving convenience and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594066000001_ABST
    Figure 0007594066000001_ABST
Patent Text Reader

Abstract

An information output system and a control program for the information output system that can improve convenience are provided. [Solution] The information output system 1 includes an image acquisition unit 101 that acquires an image, and an output unit 102 that uses the image acquired by the image acquisition unit 101 as input and outputs information identifying building materials related to the image acquired by the image acquisition unit 101 using a trained model 121 generated by machine learning.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information output system and a control program for the information output system, and more particularly to an information output system and a control program for the information output system that output information specifying building materials. [Background technology]

[0002] Building materials include wallpaper, carpets, floor sheets, tiles, stone, curtains, and decorative films. The traditional method of ordering building materials is as follows: Building material manufacturers provide catalogs with a wide variety of product photos. Purchasers of building materials, such as contractors who will be using building materials, select the desired product from the many product photos in these catalogs while imagining the product in actual use. Purchasers order products by communicating information identifying the product (such as the product number) to the manufacturer through an employee of the construction company or a building material distributor.

[0003] However, in the conventional method of ordering building materials, the purchaser had to select a product from a huge number of products listed in the catalog, which was inconvenient. Also, because there was a discrepancy between the photos in the catalog and the actual product, the purchaser was often dissatisfied because the actual product was different from what they had imagined, which was inconvenient. Furthermore, if the purchaser ordered a product sample before officially placing an order, it took time for the product sample to arrive, which was inconvenient.

[0004] Incidentally, the following Patent Document 1 discloses an online ordering system for building materials. This system is an ordering system for building materials that is carried out via communication means such as telephone lines and the Internet. In this system, an ordering menu column, an estimate menu column, a slip management menu column, and the like are provided on the display screen of the client's terminal. In this system, when an order frame in the ordering menu column is selected when placing an order, more detailed item columns are displayed. Then, ordering information such as the product code, or the product's color, wood grain, size, processing conditions, usage conditions, order quantity, and desired delivery date are entered in these item columns. In this system, the acceptability, delivery date, price, and other information corresponding to the order information are displayed on the client's terminal through computer processing on the recipient's side. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2002-269175 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, purchasers often happen to like a building material that is actually being used in a building or the like while they are out and want to purchase that building material. If purchasers could easily purchase the same building material that they actually saw and liked, the low convenience of the above-mentioned conventional method of ordering building materials could be eliminated.

[0007] However, in many cases, it is difficult for purchasers, construction companies, or building material distributors to identify the building materials they need. This is because information identifying the building materials is not displayed on the building materials after they have been applied. For example, in the case of wallpaper, the product number is displayed on the back side, which is the side that is attached to the wall, and cannot be seen from the front side. In addition, there are many building material products on the market that differ slightly in overall brightness, pattern color, size of unevenness, and shape, so it is not easy to visually distinguish between many similar products.

[0008] Therefore, buyers, construction companies, and building material dealers had to visually search for the building materials they needed from the vast number of products listed in the catalog. As a result, they often were unable to find the products they needed. Generally, building material catalogs are updated to new versions on a short cycle of about once every two years, and in the new versions, 20 to 40 percent of the products listed in the old catalog tend to be discontinued. In such a situation, the high possibility that the building materials they needed were not listed in the latest version of the catalog was one of the reasons why they were unable to find the products they needed.

[0009] Convenience remained low, as buyers, construction companies, or building material dealers had to make a lot of effort to find the products they needed.

[0010] FIG. 14 shows photographs of four different products available on the wallpaper market.

[0011] Wallpaper, an example of a building material, is available on the market in a large number of products that differ slightly in overall brightness, pattern color, size of unevenness, shape, etc. Figures 14(a) to 14(d) each show four different wallpaper products available on the market. However, it is not easy to distinguish between them visually.

[0012] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide an information output system and a control program for the information output system that can improve convenience. [Means for solving the problem]

[0013] An information output system according to one aspect of the present invention is an information output system that outputs information that identifies a building material, and includes an image acquisition unit that acquires an image, and an output unit that uses the image acquired by the image acquisition unit as an input and uses a trained model generated by machine learning to output information that identifies a building material related to the image acquired by the image acquisition unit. The information identifying the building material includes at least one of information identifying the product of the building material, a method for purchasing the building material, and a uniform resource locator (URL) of a website selling the building material.

[0014] In the above-mentioned information output system, preferably, the trained model is constructed by supervised learning in which each of a plurality of different sample images relating to a single building material is used as question data, and information identifying the single building material is used as answer data.

[0015] In the above-mentioned information output system, preferably, the multiple sample images relating to a building material include a first illumination image which is an image of the building material photographed while a first light is irradiated onto the building material, and a second illumination image which is an image of the building material photographed while a second light having a color temperature different from the color temperature of the first light is irradiated onto the building material.

[0016] In the above-mentioned information output system, preferably, the multiple sample images relating to one building material include a first cut-out image obtained by cutting out a first portion from an image photographed of the one building material, and a second cut-out image obtained by cutting out a second portion, which is different from the first portion, from the image photographed of the one building material.

[0017] In the above information output system, preferably, the proportion of the first cut-out image in an image of a single building material is greater than the proportion of the second cut-out image in an image of the single building material, and the number of pixels of the first cut-out image is the same as the number of pixels of the second cut-out image.

[0019] In the above information output system, preferably, the trained model includes a neural network.

[0020] In the above-mentioned information output system, preferably, the information output system further includes a photographing device, the photographing device including a photographing unit that photographs an image, a transmitting unit that transmits the image photographed by the photographing unit to the image acquisition unit, and a receiving unit that receives the information output by the output unit.

[0021] A control program for an information output system according to another aspect of the present invention is a control program for an information output system that outputs information identifying a building material that is wallpaper, and causes a computer to execute an image acquisition step of acquiring an image, and an output step of outputting information identifying a building material related to the image acquired by the image acquisition unit using an input of the image acquired by the image acquisition step and a trained model generated by machine learning. The information identifying the building material includes at least one of information identifying the product of the building material, a method for purchasing the building material, and a uniform resource locator (URL) of a website selling the building material. Effect of the Invention

[0022] According to the present invention, it is possible to provide an information output system and a control program for the information output system that can improve convenience. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram conceptually showing a configuration of an information output system 1 according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing a functional configuration of an information output system 1 according to an embodiment of the present invention. [Diagram 3] FIG. 5 is a diagram showing a schematic diagram of a neural network 500 of each of models 121 and 221 in one embodiment of the present invention. [Figure 4] 4 is a flowchart illustrating an operation of the information output system 1 according to the embodiment of the present invention. [Diagram 5]1 is a first diagram showing an operation screen displayed on operation display unit 303 of mobile terminal 300 in one embodiment of the present invention. FIG. [Figure 6] FIG. 2 is a second diagram showing an operation screen displayed on operation display unit 303 of mobile terminal 300 in one embodiment of the present invention. [Figure 7] FIG. 11 is a third diagram showing an operation screen displayed on operation display unit 303 of mobile terminal 300 in one embodiment of the present invention. [Figure 8] FIG. 4 is a fourth diagram showing an operation screen displayed on operation display unit 303 of mobile terminal 300 in one embodiment of the present invention. [Figure 9] FIG. 6 shows the structure of a building material 601. [Figure 10] FIG. 1 is a diagram illustrating a method for creating sample images IM1 to IM3 for a building material 601 in one embodiment of the present invention. [Figure 11] 1A to 1C are diagrams illustrating a method for augmenting learning data using a sample image IM1 according to an embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing the learning data DS for the building material 601 after it has been augmented. [Figure 13] 10 is a flowchart showing the operation of the information output system 1 regarding the learning of a model 221 in an embodiment of the present invention. [Figure 14] FIG. 1 shows photographs of four different products available on the wallpaper market. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0025] [Configuration and operation of information output system]

[0026] FIG. 1 is a diagram conceptually showing a configuration of an information output system 1 according to an embodiment of the present invention.

[0027] Referring to FIG. 1, information output system 1 (one example of an information output system) in this embodiment is an information output system that outputs information that identifies building materials. The target building materials may be of any type, such as wallpaper, carpet, floor sheet, tile, stone, curtain, or decorative film. Information output system 1 includes information output device 100, learning device 200, and multiple mobile terminals 300 (one example of a photographing device). Information output device 100, learning device 200, and multiple mobile terminals 300 are connected to each other via Internet 400 and communicate with each other. Each of information output device 100, learning device 200, and multiple mobile terminals 300 is a computer, and is configured with a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), a network interface, and the like.

[0028] The information output device 100 and the learning device 200 may be integrated into one device. In this case, the models 121 and 221 may be the same.

[0029] 2 is a block diagram showing a functional configuration of information output system 1 according to an embodiment of the present invention. Each function of information output system 1 may be realized by a CPU of information output device 100, learning device 200, and mobile terminal 300 executing a program stored in a ROM of each of information output device 100, learning device 200, and mobile terminal 300.

[0030] 2, the information output device 100 outputs information for identifying a building material. The information output device 100 includes an image acquisition unit 101 (an example of an image acquisition unit), an output unit 102 (an example of an output unit), a transmission / reception unit 103, an operation display unit 104, and a storage unit 105.

[0031] The image acquisition unit 101 acquires images of the building materials to be analyzed from a mobile terminal 300 or the like.

[0032] The output unit 102 receives the image acquired by the image acquisition unit 101 as an input, and uses the model 121 to output information that identifies the building material associated with the image acquired by the image acquisition unit 101 .

[0033] The transmitting / receiving unit 103 transmits and receives data between the learning device 200 and the mobile terminal 300 .

[0034] An operation display unit 104 accepts various operations and displays various information.

[0035] The storage unit 105 stores various information such as a model 121 (an example of a trained model).

[0036] The model 121 analyzes the contents of the input image and outputs information for identifying building materials as the analysis result. The model 121 is a trained model generated by machine learning. In this embodiment, the model 121 is generated by supervised learning and includes a neural network and parameters. The model 121 is synchronized with the model 221 of the master learning device 200 at the necessary timing so that the model 121 has the same contents as the model 221 of the master learning device 200.

[0037] The learning device 200 learns a learning model. The learning device 200 includes a learning data acquisition unit 201, a learning data augmentation unit 202, a learning unit 203, a transmission / reception unit 204, and a storage unit 205.

[0038] The learning data acquisition unit 201 acquires data used for learning the model 221. The learning data acquisition unit 201 may acquire data from a data server (not shown) via the Internet 400, or may acquire data from an external storage medium connectable to the learning device 200.

[0039] The learning data augmentation unit 202 augments data used for learning the model 221. The learning data augmentation unit 202 performs processing such as size conversion, trimming, or inversion on the data acquired by the learning data acquisition unit 201 to create augmented data.

[0040] The learning unit 203 performs learning on the model 221 .

[0041] The transmitting / receiving unit 204 transmits and receives data to and from the information output device 100 .

[0042] The storage unit 205 stores various information such as a model 221 .

[0043] The storage unit 205 may further store training data 222. The training data 222 is data used for training the model 221, and includes data acquired by the training data acquisition unit 201 and data augmented by the training data augmentation unit 202.

[0044] Each of the multiple mobile terminals 300 is owned by a respective one of the multiple users. The users are purchasers of building materials such as contractors of construction using the building materials, employees of construction companies, or vendors of building materials. Each of the multiple mobile terminals 300 has substantially the same configuration, and includes an image capturing unit 301 (an example of an image capturing unit), a transmission / reception unit 302 (an example of a transmission unit and a reception unit), an operation display unit 303, and a storage unit 304.

[0045] The photographing unit 301 photographs various images of building materials and the like.

[0046] The transmission / reception unit 302 transmits and receives data to and from the information output device 100. The transmission / reception unit 302 transmits an image captured by the imaging unit 301 to the image acquisition unit 101 of the information output device 100, and receives information output by the output unit 102 of the information output device 100.

[0047] The operation display unit 303 accepts various operations and displays various information.

[0048] The storage unit 304 stores various information such as an application 321. The application 321 is an application including a program required when the user of the mobile terminal 300 uses a service provided by the information output system 1.

[0049] Next, the neural networks of the models 121 and 221 will be described.

[0050] FIG. 3 is a diagram showing a schematic diagram of a neural network 500 of each of the models 121 and 221 in one embodiment of the present invention.

[0051] 3, the neural network 500 of each of the models 121 and 221 is a so-called hierarchical neural network, in which a large number of artificial neurons (shown as circles in FIG. 3) are connected to form a hierarchy. The hierarchical neural network includes an input artificial neuron, a processing artificial neuron, and an output artificial neuron. The neural network 500 is preferably a convolutional neural network such as an "Efficient Net."

[0052] Problem data 510 is the target of processing by the neural network 500. The problem data 510 is acquired by input artificial neurons in the input layer 501. The input artificial neurons are arranged in parallel to form the input layer 501. The problem data 510 is distributed to the processing artificial neurons.

[0053] The processing artificial neurons are connected to the input artificial neurons. The processing artificial neurons are arranged in parallel to form an intermediate layer 502. The intermediate layer 502 may have multiple layers. A neural network with three or more layers including the intermediate layer 502 is called a deep neural network.

[0054] The neural network may be a so-called convolutional neural network. A convolutional neural network is a deep neural network that is configured by alternatingly connecting convolutional layers and pooling layers. By sequentially processing the convolutional layer and the pooling layer, the image of the problem data 510 is reduced while retaining features such as edges. When the convolutional neural network is applied to image analysis, it is possible to classify images with high accuracy based on the extracted features.

[0055] The output artificial neuron outputs the correct answer data 511 to the outside. The output artificial neuron constitutes the output layer 503. The learning unit 203 learns the neural network 500 (specifically, adjusts the parameters of the model 221) so that the correct answer data 511 is output when question data 510 is input.

[0056] FIG. 4 is a flowchart illustrating the operation of the information output system 1 in one embodiment of the present invention.

[0057] 4, the mobile terminal 300 photographs a building material (S1) and transmits the photographed image to the information output device 100 (S3). In step S1, it is preferable that the building material is photographed at a position a predetermined distance (for example, 20 cm to 40 cm) away from the building material.

[0058] When the information output device 100 acquires (receives) an image from the mobile terminal 300 (S5), the acquired image is used as an input, and the information identifying a building material related to the acquired image is output (transmitted) to the mobile terminal 300 using the model 121 (S9). In step S9, the information output device 100 may output information identifying each of a plurality of possible building materials. In this case, the information output device 100 may also output the order of likelihood, the probability of accuracy, and the like for each of the identified plurality of building materials.

[0059] When the mobile terminal 300 receives information from the information output device 100 (S11), it displays the received information (S13) and ends the process.

[0060] Instead of acquiring images from the mobile terminal 300, the information output device 100 may acquire images from an external terminal (not shown) via the Internet 400, or may acquire images from an external storage medium or digital camera connectable to the information output device 100. Instead of transmitting information specifying building materials related to the acquired image to the mobile terminal 300, the information output device 100 may transmit the information to an external terminal (not shown) via the Internet 400, or may output (display) the information on the operation display unit 104 of the device itself.

[0061] [Display and operation on mobile devices]

[0062] Next, the display and operation on the mobile terminal 300 will be described.

[0063] 5 to 8 are diagrams showing operation screens displayed on operation display unit 303 of mobile terminal 300 in one embodiment of the present invention.

[0064] 5, when the user activates the application 321 (FIG. 1), the mobile terminal 300 displays an operation screen including a message M1, a camera monitor M2, and a key KY1 on the operation display unit 303. The message M1 is a message instructing the user to take a picture from a position a predetermined distance (for example, 20 cm to 40 cm) away from the building material. The key KY1 is a key that is pressed when taking a picture.

[0065] The user points the camera of the mobile terminal 300 at the building material to be analyzed at a position a predetermined distance away from the building material. The user checks the image of the building material on the monitor M2 and presses the key KY1.

[0066] 6, when key KY1 is pressed, mobile terminal 300 displays an operation screen including message M3, image M4, and keys KY2 and KY3 on operation display unit 303. Message M3 is a message inquiring whether or not to transmit a captured image to information output device 100. Image M4 is an image captured by pressing key KY1. Key KY2 is a key pressed when transmitting. Key KY3 is a key pressed when canceling transmission.

[0067] The user checks image M4 and presses key KY2 or KY3. When key KY2 is pressed, mobile terminal 300 transmits the captured image to information output device 100 and displays the operation screen shown in Fig. 7 on operation display unit 303. When key KY3 is pressed, mobile terminal 300 displays the operation screen shown in Fig. 5 on operation display unit 303.

[0068] 7, when the key KY2 is pressed, the mobile terminal 300 displays an operation screen including a message M5 indicating that the image has been transmitted and that the image is being analyzed by the information output device 100 on the operation display unit 303. Until information identifying the building material is received from the information output device 100, the mobile terminal 300 continues to display the operation screen shown in FIG.

[0069] Referring to Figure 8, when information identifying building materials is received from the information output device 100, the mobile terminal 300 displays an operation screen on the operation display unit 303, which includes a message M6 indicating that the building materials have been identified, information M7 identifying the building materials, and a key KY4.

[0070] In the operation screen shown in Fig. 8, information identifying a plurality of (three in this case) building materials is displayed as information M7 identifying a building material, arranged from top to bottom in order of likelihood. Each piece of information identifying a building material includes the product number of the building material, the manufacturer of the building material, and the address of a website where the building material can be purchased. When key KY4 is pressed, the mobile terminal 300 displays the operation screen shown in Fig. 5 on the operation display unit 303.

[0071] [How building materials appear in the captured images]

[0072] Next, how building materials appear in a captured image will be described, assuming that the building material to be analyzed is wallpaper.

[0073] Fig. 9 is a diagram showing the structure of a building material 601. Fig. 9(a) is a plan view, and Fig. 9(b) is a cross-sectional view taken along line IXB-IXB in Fig. 9(a), which is a cross-sectional view of a row 611 of patterns in the building material 601. Fig. 9(c) is a cross-sectional view taken along line IXC-IXC in Fig. 9(a), which is a cross-sectional view of a row 612 of patterns in the building material 601. The rows 612 and 611 are adjacent to each other and are arranged alternately in the vertical direction.

[0074] Referring to FIG. 9, regardless of the type of pattern of a building material, the surface of the building material generally has unevenness, and a shadow is generated due to the unevenness. Building material 601 is an example of a building material, and has a checkered pattern. Of two adjacent quadrangles that form the checkered pattern, one forms a convex portion 613, and the other forms a concave portion 614. A shadow is generated due to the unevenness in a part of the periphery of convex portion 613. Therefore, the position where the shadow is generated depends on the position of the lighting that irradiates light onto building material 601. The length and darkness of the shadow depend on the intensity of the lighting light and the degree of the unevenness.

[0075] Furthermore, the color of the surface of the building material 601 depends on the color temperature of the lighting. For example, if the lighting is what is known as "warm white," the surface of the building material 601 will have a warm color. If the lighting is what is known as "daylight white," the surface of the building material 601 will have a color close to its natural state. If the lighting is what is known as "daylight," the surface of the building material 601 will have a cool color.

[0076] Furthermore, when the building material 601 is close, the unevenness and shadows of the building material 601 are clearly visible, but when the building material is close, the unevenness and shadows become difficult to see.

[0077] Furthermore, the appearance of the building material in the captured image will vary depending on the performance of the camera used to capture the image and the part of the building material that is captured.

[0078] Therefore, even for the same building material, the appearance of the building material in a photographed image changes depending on various conditions such as the position of the lighting, the intensity of the lighting, the level of unevenness, the color temperature of the lighting, the distance to the building material, the performance of the camera, the part of the building material that is photographed, etc. Such changes in appearance due to various conditions can occur not only with wallpaper, but with building materials in general.

[0079] [How the learning unit trains the model]

[0080] As described above, the appearance of the building materials in the captured image changes depending on various conditions. The change in the appearance of the building materials in the captured image reduces the accuracy of the information identifying the building materials output by the information output system.

[0081] Therefore, it is preferable that the learning unit 203 constructs the model 221 by supervised learning, in which each of a plurality of different sample images relating to one building material is used as question data, and information identifying one building material is used as answer data. As a result, the model 121 of the information output device 100 learns changes in the appearance of building materials depending on different conditions, and the accuracy of the information identifying the building material output by the information output system can be improved.

[0082] Next, an example of a method for learning the model 221 performed by the learning unit 203 will be described.

[0083] Fig. 10 is a diagram for explaining a method for creating sample images IM1 to IM3 of a building material 601 in one embodiment of the present invention. In Fig. 10 and Fig. 12, differences in the color of the building material caused by differences in the color temperature of the lighting are expressed by different types of thin hatching. Also, shadows that appear around convex parts are expressed by thick hatching, assuming that light is irradiated from the upper left direction of the building material. These hatchings make it clear that the appearance of the building material 601 in each of the sample images IM1 to IM3 changes depending on various conditions.

[0084] With reference to Fig. 10, a method for creating an input data ID for one building material 601 among a plurality of building materials will be described here. The administrator of the information output system 1 fixes a light at a predetermined position (here, the upper left position of the building material). A sample image of the building material 601 is taken at a position a predetermined distance away from the building material 601. The distance at which the sample image is taken is preferably equal to the distance from the mobile terminal 300 to the building material when the building material is photographed with the mobile terminal 300 when the information output system 1 is used.

[0085] When taking the sample images, the manager takes each of the multiple sample images of the building material 601 while irradiating the building material with each of multiple lights having different color temperatures. Here, sample image IM1 is taken while irradiating the building material 601 with incandescent light, sample image IM2 is taken while irradiating the building material 601 with neutral white light, and sample image IM3 is taken while irradiating the building material 601 with daylight light. As a result, three mutually different sample images IM1 to IM3 (examples of first and second illumination images) of the building material 601 are obtained.

[0086] The administrator inputs sample images IM1 to IM3 related to the building material 601 and information IF1 that identifies the building material 601 to the learning device 200 as input data ID related to the building material 601. The input data ID related to the building material 601 becomes part of the learning data DS (FIG. 12) related to the building material 601. It is preferable that the information IF1 includes at least one of the product number of the building material, the method of purchasing the building material, and the URL of a website that sells the building material. The number of sample images included in the input data ID is arbitrary, and it is preferable that there is more than one.

[0087] In addition, the administrator also creates input data IDs for the other building materials 602, 603, . . . of the multiple building materials in a similar manner to that for the building material 601, and inputs these to the learning device 200.

[0088] When the learning device 200 acquires the input data ID related to the building material 601, it augments the learning data related to the building material 601 at the necessary timing. It is preferable that the learning data is augmented using at least one of the processing methods of color conversion, contrast change, noise addition, size conversion, trimming, and inversion for the sample image. Here, the learning data is augmented by the following method.

[0089] FIG. 11 is a diagram illustrating a method for augmenting learning data using a sample image IM1 in one embodiment of the present invention.

[0090] 11(a), the learning device 200 obtains each of the cut-out images CM11 to CM17 (examples of the first and second cut-out images) by cutting out a plurality of (seven here) different portions of the sample image IM1. The proportions of the cut-out images CM11 to CM13 (examples of the first cut-out image) in the sample image IM1 are equal to each other, and the number of pixels of the cut-out images CM11 to CM13 are equal to each other. The proportions of the cut-out images CM14 to CM17 (examples of the second cut-out image) in the sample image IM1 are equal to each other, and the number of pixels of the cut-out images CM14 to CM17 are equal to each other. The proportions of the cut-out images CM11 to CM13 in the sample image IM1 are greater than the proportions of the cut-out images CM14 to CM17 in the sample image IM1.

[0091] 11(b), each of the smaller cut-out images CM14-CM17 is used as each of the sample images IM14-IM17 of the building material 601 with the same number of pixels. Meanwhile, each of the larger cut-out images CM11-CM13 is reduced in number of pixels to be the same as each of the cut-out images CM14-CM17 (in other words, each of the cut-out images CM11-CM13 is processed into a coarse image) to be each of the sample images IM11-IM13. In this manner, sample image IM1 is processed to obtain sample images IM11-IM17. Note that the number of sample images created by processing one sample image is arbitrary.

[0092] FIG. 12 is a diagram showing the learning data DS regarding the building material 601 after the augmentation.

[0093] 12, the learning device 200 processes each of the sample images IM2 and IM3 in a similar manner to sample image IM1 to obtain each of the sample images IM21-IM27 and IM31-37. Each of the sample images IM21-IM27 is created by processing sample image IM2, and corresponds to each of the sample images IM11-IM17. Each of the sample images IM31-IM37 is created by processing sample image IM3, and corresponds to each of the sample images IM11-IM17.

[0094] The learning device 200 adds sample images IM1, IM11-IM17, IM2, IM21-27, and IM3, IM31-IM37 (examples of multiple sample images) and information IF1 to the learning data 222 as learning data DS related to the building material 601. The sample images IMI, IM11-IM17, IM2, IM21-27, and IM3, IM31-IM37 are set as problem data of the learning data DS, and the information IF1 is set as answer data of the learning data DS. In this way, the learning data DS related to the building material 601 is augmented by the sample images IM11-IM17, IM21-27, and IM31-IM37.

[0095] FIG. 13 is a flowchart showing the operation of the information output system 1 regarding the learning of the model 221 in one embodiment of the present invention.

[0096] 13, when learning device 200 acquires input data IDs related to n (n is a natural number) building materials (S101), it sets variable k to 1 (S103) and extracts the input data ID related to the kth building material (S105). Next, learning device 200 uses the extracted input data to augment the learning data related to the kth building material (S107) and adds it to learning data 222 (S109). Finally, learning device 200 determines whether variable k is equal to n (S111).

[0097] In step S111, if it is determined that the variable k is not equal to n (NO in S111), the learning device 200 increments the variable k (S113) and proceeds to the process of step S105.

[0098] In step S111, if it is determined that the variable k is equal to n (YES in S111), all of the learning data for the n building materials has been added to the learning data 222. In this case, the learning device 200 uses the learning data 222 to learn its own model 221 (S115). Thereafter, the learning device 200 transmits the data of its own model 221 to the information output device 100 (S117). When the information output device 100 receives the data of the model 221 of the learning device 200 (S119), it synchronizes the model 121 with the model 221 of the learning device 200 so that the content of the model 121 of the own device is the same as that of the model 221 of the learning device 200 (S121), and ends the process.

[0099] [Effects of the embodiment]

[0100] In this embodiment, a user, such as a purchaser, a construction company, or a building material dealer, can obtain information for identifying a required building material by taking an image of the required building material. This eliminates the need to visually identify the required building material from among the vast number of products listed in the catalog, improving convenience. In addition, even if the required building material is not listed in the latest version of the catalog, the required product can be found, improving convenience.

[0101] In addition, a trained model is constructed by supervised learning in which each of a plurality of different sample images of one building material is used as question data and information identifying the one building material is used as answer data. As a result, the trained model has learned how the appearance of the building material changes depending on different conditions, and the accuracy of the information identifying the building material can be improved.

[0102] In addition, the multiple sample images of a certain building material include multiple illumination images taken while irradiating the certain building material with multiple lights having different color temperatures, respectively. This allows the trained model to learn how the appearance of the certain building material changes depending on the color temperature of the illumination light, thereby improving the accuracy of the information identifying the building material.

[0103] In addition, the multiple sample images of one building material include multiple cut-out images obtained by cutting out multiple different parts of an image of one building material. This allows the trained model to learn how the appearance of the building material changes depending on the part of the building material that is photographed, thereby improving the accuracy of the information identifying the building material.

[0104] Furthermore, the ratio of the first cut-out image to the image of the one building material is greater than the ratio of the second cut-out image to the image of the one building material, and the number of pixels of the first cut-out image is the same as the number of pixels of the second cut-out image. This allows the trained model to learn how the appearance of the building material changes depending on the distance to the building material, thereby improving the accuracy of the information identifying the building material.

[0105] [others]

[0106] Each of the models 121 and 221 may be generated by unsupervised learning. When each of the models 121 and 221 is generated by unsupervised learning, each of the models 121 and 221 may be generated by machine learning based on a plurality of different sample images related to building materials, not based on information identifying the building materials (in other words, machine learning without correct answer data). Each of the models 121 and 221 may be any learning model, and may include a neural network only in a part such as Vision Transformer (ViT), or may not include a neural network.

[0107] In the above embodiment, an example has been described in which the manager fixes the lighting at a predetermined position (the upper left position of the building material) and captures each of the multiple sample images IM1 to IM3 of the building material 601 while irradiating each of multiple lights having different color temperatures. In addition to or instead of this example, the manager may capture each of the multiple different sample images of the building material 601 while placing lighting at each of multiple different positions. This allows each of the models 121 and 221 to learn based on the multiple sample images captured while the building material 601 is irradiated with the light of the lighting from various angles, and the accuracy of the information identifying the building material can be improved.

[0108] The processes in the above-described embodiments and modifications may be performed by software or by using hardware circuits. A program for executing the processes in the above-described embodiments may be provided, or the program may be recorded on a recording medium such as a CD-ROM, a flexible disk, a hard disk, a ROM, a RAM, or a memory card and provided to a user. The program is executed by a computer such as a CPU. The program may be downloaded to the device via a communication line such as the Internet.

[0109] The above-described embodiments and modifications can be combined as appropriate.

[0110] The above-described embodiments and modifications should be considered as illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0111] 1. Information output system (an example of an information output system) 100 Information output device 101 Image acquisition unit (an example of an image acquisition unit) 102 Output section (an example of an output section) 103, 204, 302 Transmitter / receiver (an example of a transmitter and receiver) 104,303 Operation display section 105,205,304 Storage section 121,221 model (an example of a trained model) 200 Learning Device 201 Learning Data Acquisition Unit 202 Learning Data Augmentation Unit 203 Learning Department 222,DS Learning Data 300 Mobile terminal (an example of a photographing device) 301 Photography department (example of photography department) 321 Apps 400 Internet 500 Neural Networks 501 Input Layer 502 Middle Class 503 Output Layer 510 Problem Data 511 Correct Data 601,602,603 ​​Building materials 611,612 Patterned rows in building materials 613 Convex parts in building materials 614 Recesses in Building Materials CM11 to CM17 Cutout images (examples of the first and second cutout images) ID Input Data IF1 Information identifying building materials IM1-IM3, IM11-IM17, IM21-IM27, IM31-IM37 Sample images (examples of multiple sample images and first and second illumination images) KY1~KY4 keys M1, M3, M5, M6 Messages M2 Monitor M4 Images M7 Information

Claims

1. An information output system that outputs information identifying a building material, an image acquisition unit that acquires an image; and an output unit that uses an image acquired by the image acquisition unit as an input and a trained model generated by machine learning to output information identifying a building material associated with the image acquired by the image acquisition unit, An information output system, wherein the information specifying the building material includes at least one of information specifying a product of the building material, a method for purchasing the building material, and a URL (Uniform Resource Locator) of a website selling the building material.

2. The information output system of claim 1, wherein the trained model is constructed by supervised learning in which each of a plurality of different sample images relating to a single building material is used as question data, and information identifying the single building material is used as answer data.

3. The information output system of claim 2, wherein the plurality of sample images relating to the one building material include a first illumination image which is an image of the one building material photographed while irradiating the one building material with a first light, and a second illumination image which is an image of the one building material photographed while irradiating the one building material with a second light having a color temperature different from the color temperature of the first light.

4. The information output system of claim 3, wherein the plurality of sample images relating to the one building material include a first cut-out image obtained by cutting out a first portion from an image photographed of the one building material, and a second cut-out image obtained by cutting out a second portion, which is different from the first portion, from the image photographed of the one building material.

5. The information output system of claim 4, wherein the proportion of the first cut-out image in the image of the one building material is greater than the proportion of the second cut-out image in the image of the one building material, and the number of pixels of the first cut-out image is the same as the number of pixels of the second cut-out image.

6. The information output system according to claim 1 , wherein the trained model includes a neural network.

7. The information output system further includes an imaging device, The imaging device is An image capturing unit that captures an image; a transmission unit that transmits the image captured by the imaging unit to the image acquisition unit; The information output system according to claim 1 , further comprising a receiving section for receiving the information output by said output section.

8. A control program for an information output system that outputs information identifying a building material, an image acquisition step of acquiring an image; and an output step of outputting information for identifying a building material associated with the image acquired in the image acquisition step using a trained model generated by machine learning, using the image acquired in the image acquisition step as an input. A control program for an information output system, wherein the information identifying the building material includes at least one of information identifying the product of the building material, a method for purchasing the building material, and a URL (Uniform Resource Locator) of a website selling the building material.

Citation Information

Patent Citations

  • Image processing system and sense information processing system

    JP2002216133A

  • System for placing and receiving order for building materials

    JP2002269175A

  • Situation determination system, situation determination method, and situation determination program

    JP2017107443A

  • Data generation device, learned model generation device, weighing machine and data generation method

    JP2021128388A

  • Information processing apparatus, inference model construction method, information processing method, inference model, program, and recording medium

    JP2021140445A