Image recognition device and image recognition method

Abstract

An image identification device (10) comprises: an image acquisition unit (110) that acquires an image including first members; a setting unit (141) that sets one or more regions in the image; a first determination unit (142) that determines, for each of the one or more regions in the image, whether a joining point is included between two of the first members; and an identification unit (144) that, by using a first identification model constructed on the basis of a first training image that does not include a joining point between two members, the training image being of a plurality of different types of members including first members, receives input of a region, from among the one or more regions in the image, that the first determination unit (142) has determined to not include a joining point between two of the first members, identifies the type of the first members, and outputs type information indicating the type of the first members.

Description

[Technical Field]

[0001] The present disclosure relates to an image recognition device and an image recognition method. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for identifying (specifying) the characteristics of a component by processing an image obtained by photographing the component (see, for example, Patent Documents 1 and 2).

[0003] Patent Documents 1 and 2 disclose devices for determining the color and pattern of objects such as pieces of cloth, architecture, furniture, interiors (interior materials), clothing, and printed matter. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6197521 [Patent Document 2] Japanese Patent Application Publication No. 2018-189420 Summary of the Invention [Problem to be solved by the invention]

[0005] According to conventional devices, by identifying the color, pattern, etc. of the components included in an image, it is possible to identify (specify) the type of component included in the image from among a plurality of components that have different types of color, pattern, etc. It is desirable for this type of device to have improved accuracy in identifying components included in an image.

[0006] The present disclosure provides an image recognition device and the like with improved recognition accuracy. [Means for solving the problem]

[0007] An image identification device according to one embodiment of the present disclosure includes an image acquisition unit that acquires an image including a first member, a setting unit that sets one or more regions in the image, a first determination unit that determines whether each of the one or more regions in the image includes a seam between two of the first members, and an identification unit that uses a first identification model constructed based on first training images of each of a plurality of different types of members including the first member that do not include a seam between the two members, to identify the type of the first member by using as input an area of ​​the one or more regions in the image that the first determination unit determines does not include a seam between the two first members, and outputs type information indicating the type of the first member.

[0008] An image identification method according to one embodiment of the present disclosure acquires an image including a first member, sets one or more regions in the image, determines whether each of the one or more regions in the image includes a seam between two of the first members, and uses a first identification model constructed based on first training images of a plurality of different types of members including the first member that do not include a seam between the two members to identify the type of the first member using, as input, a region of the one or more regions in the image that is determined not to include a seam between the two first members, and outputs type information indicating the type of the first member.

[0009] A program according to one aspect of the present disclosure is a program for causing a computer to execute the image identification method. [Effects of the Invention]

[0010] According to the present disclosure, an image recognition device or the like with improved recognition accuracy is realized. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining an overview of an image recognition device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a display screen of a classification result of the image classification device according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating a functional configuration of the image identification device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of an image acquired by the image identification device according to the embodiment and a set area. [Figure 5] FIG. 5 is a diagram illustrating a joint between two members according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining the area resetting process performed by the image identification device according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating the operation procedure of the image identification device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Summary of the Disclosure) As described above, conventional devices can identify (also called distinguishing or identifying) the color, pattern, etc. of components contained in an image, making it possible to identify the type of component contained in the image from among multiple components that have different types of color, pattern, etc.

[0013] For example, the device disclosed in Patent Document 1 identifies whether a piece of fabric is "plain" or "patterned" from the amplitude of the color value waveform, and if it is "plain," further identifies whether the piece of fabric is "plain white" or "plain color."

[0014] Furthermore, for example, the device disclosed in Patent Document 2 includes a shielding means for optically shielding the surface of the object, the lighting means, and the imaging means from the outside. The device disclosed in Patent Document 2 identifies the category (type) of the surface of the object using statistical processing from multiple feature amounts of images captured by switching between multiple lighting patterns.

[0015] Incidentally, for example, interior materials for buildings are used by joining together a plurality of components (unit components). Therefore, an image generated by photographing the components in this joined state may be used to identify the type of the components. In such cases, if the image used to identify the components (unit components) contains a seam located between two or more (e.g., two) components formed when the components are joined together, misidentification may occur (i.e., an incorrect identification result may result). In other words, if the image to be processed contains a seam, the accuracy of identifying the type of component may decrease.

[0016] For example, to identify the type of component, a discrimination model for discriminating the type of component is used, which is trained (machine learned) using only images of components (unit components) that do not include seams as images (image data) used in the training data. When such a discrimination model is used, the above-mentioned misidentification is particularly likely to occur.

[0017] On the other hand, forcing a user to photograph a component without including the seam would make the device for identifying components less user-friendly.

[0018] Therefore, the inventors of the present application discovered that even when an image contains seams, it is possible to improve the accuracy of identifying the type of component contained in the image by using areas of the image that do not contain seams.

[0019] An image identification device according to one embodiment of the present disclosure includes an image acquisition unit that acquires an image including a first member, a setting unit that sets one or more regions in the image, a first determination unit that determines whether each of the one or more regions in the image includes a seam between two of the first members, and an identification unit that uses a first identification model constructed based on first training images of each of a plurality of different types of members including the first member that do not include a seam between the two members, to identify the type of the first member by using as input an area of ​​the one or more regions in the image that the first determination unit determines does not include a seam between the two first members, and outputs type information indicating the type of the first member.

[0020] Also, for example, the image identification device further includes a second judgment unit that judges whether the number of areas that the first judgment unit has judged not to include a seam between two of the first components among the one or more areas is equal to or greater than a predetermined number, and when the second judgment unit judges that the number is less than the predetermined number, the setting unit further sets an area that is at least partially different from each of the one or more areas, and the first judgment unit further judges whether the area further set by the setting unit includes a seam between two of the first components.

[0021] Also, for example, the setting unit further resets an area that is at least partially different from the area that the first determination unit determined to include a seam between the two first members, and the first determination unit further determines whether the area reset by the setting unit includes a seam between the two first members.

[0022] Also, for example, in the resetting, the setting unit resets the region based on the position of a seam between the two first members.

[0023] Also, for example, in the resetting, the setting unit resets the area by moving the area that the first determination unit has determined to include a seam between two of the first members in a direction intersecting the longitudinal direction of the seam.

[0024] Furthermore, for example, the first determination unit determines whether or not a joint between two of the first members is included by detecting a straight line included in the image.

[0025] Also, for example, the first judgment unit uses a second identification model constructed based on a plurality of the first training images and a second training image including a seam between two of a plurality of different types of components including the first component, to input the one or more regions in the image and output seam information indicating whether or not each of the one or more regions includes a seam between the two first components, thereby determining whether or not each of the one or more regions includes a seam between the two first components.

[0026] Furthermore, for example, each of the plurality of components is an interior material to be applied to a building.

[0027] Also, for example, the image identification device further includes a display unit that displays an image indicating the type of the first member indicated by the type information output by the identification unit.

[0028] Furthermore, an image identification method according to one embodiment of the present disclosure acquires an image including a first member, sets one or more regions in the image, determines whether each of the one or more regions in the image includes a seam between two of the first members, and uses a first identification model constructed based on first training images of a plurality of members of different types including the first member that do not include a seam between the two members as input to identify the type of the first member using a region of the one or more regions in the image that is determined not to include a seam between the two first members, and outputs type information indicating the type of the first member.

[0029] Furthermore, a program according to one aspect of the present disclosure is a program for causing a computer to execute the image identification method.

[0030] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0031] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, step order, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims will be described as optional components.

[0032] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0033] (Embodiment) [overview] First, an overview of an image identification device 10 according to an embodiment will be described. Fig. 1 is a diagram for explaining the overview of an image identification device 10 according to an embodiment.

[0034] The image recognition device 10 is realized, for example, in the form of a tablet terminal, and is used by a user who inspects interior materials. In this case, inspecting interior materials means inspecting whether the interior materials are installed according to specifications.

[0035] For example, in the sale of newly built condominiums, many options are available for interior materials such as flooring and wall coverings to accommodate various customer preferences. Therefore, before the residence is handed over to the customer, it is necessary to inspect whether the interior materials specified by the customer have been installed correctly. The image recognition device 10 is used to inspect such interior materials.

[0036] Note that interior materials are a general term for the finishing and underlayment materials used on floors, walls, ceilings, etc. Interior materials include not only the finishing materials that directly face the interior of the room, such as flooring, carpet, tiles, wall coverings, plywood, and paint, but also the underlayment materials underneath.

[0037] When the image recognition device 10 acquires an image of an interior material through a user operation, it recognizes whether the interior material shown in (included in) the image is an interior material that meets specifications, and displays the recognition result on the display unit 130.

[0038] FIG. 2 is a diagram showing an example of a display screen of the classification result of the image classification device 10 according to the embodiment.

[0039] The image identification device 10 outputs, for example, information (type information) indicating the type of component included in the image. "Flooring material A" shown in FIG. 2 is an example of type information, and indicates that the type of component included in the image is "Flooring material A." As shown in FIG. 2, for example, the display unit 150 displays the type information (more specifically, an image of characters or the like indicating the type of component indicated by the type information) and an image of the component indicated by the type information.

[0040] The image of the component displayed by the display unit 130 may be an image captured by the image acquisition unit 110 (see FIG. 3) (e.g., image 300 including the seam 211 shown in FIG. 4), or a sample image of the component stored in the memory unit 150 (see FIG. 3) (e.g., an image of a unit component).

[0041] [composition] Next, the configuration of the image identification device 10 will be described.

[0042] FIG. 3 is a block diagram showing the functional configuration of the image identification device 10 according to the embodiment.

[0043] The image recognition device 10 includes an image acquisition unit 110, an operation reception unit 120, a display unit 130, an information processing unit 140, and a storage unit 150.

[0044] Image acquisition unit 110 acquires an image. For example, image acquisition unit 110 acquires (i.e., captures) an image when a shooting operation is accepted by operation acceptance unit 120. The image acquired by image acquisition unit 110 includes a member (more specifically, a first member described later).

[0045] The components are not particularly limited and may be any object such as interior materials, building materials, furniture, etc. In this embodiment, the multiple components to be identified by the identification unit 144, which will be described later, are each interior materials applied to a building.

[0046] The first member is a member that is included in the image and whose type is identified (i.e., whose type is specified) by the identification unit 144, which will be described later. In other words, the first member is a member (in other words, a subject) that is included in the image acquired by the image acquisition unit 110, and is a member that the identification unit 144 identifies using an identification model.

[0047] The image acquisition unit 110 is, for example, a camera realized by a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. The image acquired by the image acquisition unit 110 is stored in the storage unit 150.

[0048] Note that the image acquisition unit 110 may be a communication interface that acquires images from a camera other than the image acquisition unit 110. Alternatively, the image acquisition unit 110 may be a processing unit that acquires images from the storage unit 150 when the images (image data) are stored in advance in the storage unit 150.

[0049] The operation reception unit 120 receives user operations and is realized by a touch panel, hardware buttons, and the like.

[0050] The display unit 130 displays a display screen under the control of the information processing unit 140. For example, the display unit 130 displays an image indicating the type of member (more specifically, the first member) indicated by the type information output by the identification unit 144. The display unit 130 is, for example, a display including a liquid crystal panel, an organic EL panel, or the like as a display device.

[0051] The information processing unit 140 performs various information processes to identify the type of material (e.g., interior materials) included in (in other words, shown in) the image acquired by the image acquisition unit 110. For example, the information processing unit 140 performs various information processes to identify the type of interior materials included in the image acquired by the image acquisition unit 110. The information processing unit 140 may perform processing to identify whether the interior materials comply with specifications by identifying the type of interior materials included in the image acquired by the image acquisition unit 110. The information processing unit 140 is realized, for example, by a microcomputer, but may also be realized by a processor.

[0052] Specifically, the information processing unit 140 includes a setting unit 141, a first determination unit 142, a second determination unit 143, and a recognition unit 144.

[0053] The setting unit 141 is a processing unit that sets one or more regions in the image acquired by the image acquisition unit 110.

[0054] FIG. 4 is a diagram showing an example of an image acquired by the image recognition device 10 according to the embodiment and a set area.

[0055] For example, suppose that the image acquisition unit 110 acquires an image 300. In this case, for example, the setting unit 141 sets one or more regions in the image 300. In the example shown in FIG. 4, the setting unit 141 sets a region 310, a region 311, a region 312, a region 313, and a region 314.

[0056] The size of the region may be determined arbitrarily in advance and is not particularly limited. For example, the region may be a part of the image or the entire image.

[0057] Furthermore, the shape of the region is rectangular (more specifically, oblong) in this embodiment, but may also be square or circular, and is not particularly limited.

[0058] The sizes and shapes of the regions may be the same or different from one another.

[0059] Furthermore, in this embodiment, the setting unit 141 sets five regions, but it is sufficient to set one or more regions, and the number of regions set by the setting unit 141 may be one, or two or more.

[0060] The setting unit 141 may set an area in any part of the image, or may set two or more areas by dividing the entire image into areas. Alternatively, the setting unit 141 may set the entire image as one area.

[0061] Furthermore, the setting unit 141 may set the area at any position in the image. For example, the setting unit 141 may set the area at a predetermined position in the image based on coordinate information indicating the position stored in the memory unit 150, or may generate a random number, determine coordinates based on the generated random number, and set the area based on the determined coordinates.

[0062] Furthermore, the setting unit 141 may set the regions so that they partially overlap, or so that they do not overlap.

[0063] The setting unit 141 sets an area in the image acquired by the image acquisition unit 110, for example, with a predetermined size, shape, and position.

[0064] In the following description, a portion of an image surrounded by a region will also be referred to as a partial image or simply as a region.

[0065] The first determination unit 142 is a processing unit that determines whether or not a seam between two first members is included in each of the one or more regions set by the setting unit 141 (i.e., in each partial image) in the image acquired by the image acquisition unit 110. In other words, the first determination unit 142 determines whether or not a seam is included in the portion surrounded by the regions in the image.

[0066] A seam between two components is a portion located between the two components. The seam may be, for example, a gap between the components, but it may also be a portion where the components meet, or a resin or the like located between the components to join them. The multiple components forming the seam may be the same type of component or different types of component. In other words, the multiple first components forming the seam may be the same type or different types.

[0067] Furthermore, the seam between two members is, for example, a seam located between two members, but it may also be a seam located in a portion surrounded by three or more members.

[0068] 5 is a diagram illustrating a joint between two members 200 according to the embodiment. Note that in FIG. 5, members 200 are hatched for ease of explanation.

[0069] As shown in Fig. 5, for example, when the member 200 is an interior material, a plurality of members 200 (unit members) shown in Fig. 5 are joined together for use. Therefore, a seam 210 is formed between two members 200.

[0070] 4, a seam 211 is located between members 201 and 202 (both of which are examples of a first member) that are both woodgrain. For example, the first determination unit 142 determines that the seam 211 is not included in regions 310 and 314. On the other hand, for example, the first determination unit 142 determines that the seam 211 is included in regions 311, 312, and 313.

[0071] Furthermore, for example, the setting unit 141 further resets an area that is at least partially different from the area that the first determination unit 142 has determined to include a seam between two or more first members.

[0072] FIG. 6 is a diagram for explaining the area resetting process performed by the image identification device 10 according to the embodiment.

[0073] 6, the region 311 includes the seam 211. In this case, the first determination unit 142 determines that the region 311 (i.e., the partial image of the region 311) includes the seam 211 between the two members 201 and 202. In this case, the setting unit 141 resets the region 311. For example, the setting unit 141 resets the region by moving the region 311 to the region 315.

[0074] For example, the first determination unit 142 further determines whether or not the region 315, which has been reset by the setting unit 141, includes a seam between two or more first members. In the example shown in Fig. 6, the first determination unit 142 determines that the region 315 does not include the seam 211 between the members 201 and 202.

[0075] In the above-described resetting (processing of resetting an area), the setting unit 141 may arbitrarily determine the position of the area. For example, the setting unit 141 may reset the area to a predetermined position in the image based on coordinate information indicating the position stored in the storage unit 150, or may generate a random number, determine coordinates based on the generated random number, and reset the area based on the determined coordinates.

[0076] Alternatively, in the resetting, the setting unit 141 may reset the region based on the position of the seam between the two first members.

[0077] For example, the setting unit 141 detects the position of a seam in an area determined by the first determination unit 142 to include a seam, or the position of a seam in the entire image, using line detection, edge detection, or the like. The setting unit 141 further resets an area in the image so as not to include a seam, based on the detection result. Alternatively, for example, the setting unit 141 may reset an area based on the position of the seam by setting an area so as not to overlap with an area determined by the first determination unit 142 to include a seam.

[0078] Alternatively, in the resetting, the setting unit 141 may reset the area by moving the area that the first determination unit 142 has determined to include a seam between two first members in a direction intersecting the longitudinal direction of the seam.

[0079] For example, in interior materials installed in buildings, the seams are often straight (linear), so it is thought that by moving the area including the seam in a direction intersecting the longitudinal direction of the seam, it will be easier to set the area excluding the seam.

[0080] The first determination unit 142 may determine whether or not a seam is included in the image (more specifically, the partial image in the area set by the setting unit 141) by any process.

[0081] For example, as mentioned above, since the seams in interior materials and the like installed in buildings are often straight lines, the first determination unit 142 may determine whether or not a seam between two first components is included by detecting straight lines included in the image.

[0082] Alternatively, for example, the first determination unit 142 may use a discriminative model (second discriminative model) trained by machine learning to determine whether or not a seam is included in the region set by the setting unit 141. That is, for example, the first determination unit 142 uses the trained second discriminative model to determine whether or not a seam is included in a partial image of the region set by the setting unit 141. For example, the first determination unit 142 uses the second discriminative model constructed based on a plurality of first training images and a plurality of second training images (second training images) of a plurality of different types of components including a first component, the second training images including a seam between the two components, as input, to output seam information indicating whether or not a seam between the two first components is included in each of the one or more regions, thereby determining whether or not a seam between the two first components is included in each of the one or more regions.

[0083] The second discrimination model is a machine learning model that determines whether an image contains a seam. Specifically, the second discrimination model is a machine learning model that is trained using, as training data, a plurality of first training images, second training images that include a seam between two members of a plurality of different types of members that include the first member, and training seam information that indicates whether a seam between the two members is included in each of the plurality of first training images and the plurality of second training images.

[0084] The first learning images are images of the plurality of members, respectively. Specifically, the first learning images are images of the plurality of members of different types, including the first member, and do not include a seam between two members.

[0085] The second learning images are images of the plurality of members, specifically, the second learning images are images of the plurality of members of different types, including the first member, and include a seam between two members.

[0086] Note that one learning image (each of the first learning image and the second learning image) may include only one component, or may include two or more components.

[0087] The training seam information is information indicating whether or not a seam between two members is included in each of the plurality of first training images and the plurality of second training images.

[0088] As described above, the second discrimination model is an inference model (discrimination model) that has been machine-trained using a plurality of first learning images, a plurality of second learning images, and learning seam information, which is teaching data (so-called annotation information), as learning data.

[0089] The second determination unit 143 is a processing unit that determines whether the number of regions that the first determination unit 142 has determined not to include seams between the multiple first members among the one or more regions is equal to or greater than a predetermined number. For example, if the second determination unit 143 determines that the number is less than the predetermined number, the setting unit 141 further sets a region that is at least partially different from each of the one or more set regions. In this case, for example, the first determination unit 142 further determines whether a seam between the multiple first members is included in the image for the region further set by the setting unit 141.

[0090] The setting unit 141 may set one region at a time, or may set two or more regions at a time. For example, the process of setting a region by the setting unit 141, the process of determining whether or not there is a seam by the first determination unit 142, and the process of determining the number of regions by the second determination unit 143 may be repeatedly executed in this order.

[0091] Furthermore, the number of regions set by the setting unit 141 may be more than, the same as, or less than the predetermined number.

[0092] The predetermined number may be 1 or may be 2 or more, and is not particularly limited.

[0093] Moreover, information indicating the predetermined number (area number information) is stored in advance in the storage unit 150, for example.

[0094] The identification unit 144 is a processing unit that uses a first identification model to identify the type of a first component included in an image and outputs type information indicating the type. Specifically, the identification unit 144 uses the first identification model to identify the type of a first component included in a region that the first determination unit 142 has determined does not include a seam between two first components. More specifically, the identification unit 144 uses a first identification model constructed based on a first learning image that does not include a seam between two components, for each of a plurality of components of different types including the first component, to identify the first component using, as input, a region (i.e., a partial image) that the first determination unit 142 has determined does not include a seam between two first components, out of one or more regions set by the setting unit 141 in the image, and outputs type information indicating the type of the first component. In other words, the identification unit 144 uses the trained first identification model to identify the type of first component that appears in the image (image data) acquired by the image acquisition unit 110 and one or more areas (area information indicating the areas) set by the setting unit 141 (more specifically, that appears in a partial image of the area).

[0095] For example, as shown in FIG. 4, if the setting unit 141 sets five regions 310 to 314 and the first judgment unit 142 determines that two of the five regions 310 to 314, 310 and 314, do not contain a seam 211, the identification unit 144 uses the first identification model to identify the types of components 201 and 202 contained in the image 300 based on the two regions 310 and 314 in the image 300 (i.e., the partial image of region 310 and the partial image of region 314), and outputs type information indicating the identification result.

[0096] The first identification model is a machine learning model that identifies the type of component included in an image. Specifically, the first identification model is a machine learning model that is trained using, as training data, a plurality of first training images of a plurality of different types of components including a first component, each of which does not include a seam between the two components, and training type information that indicates the type of component included in each of the plurality of first training images.

[0097] The learning type information is information that indicates the type of component included in each of the plurality of first learning images.

[0098] As described above, the first discrimination model is an inference model that is machine-learned using a plurality of first learning images and learning type information (so-called annotation information) that is teacher data as learning data.

[0099] The plurality of components are any objects such as the above-mentioned interior materials, building materials, furniture, etc., and are components whose type can be identified by the identification unit 144.

[0100] The type of component identified by the identification unit 144 is information for identifying the component, such as a predetermined component product number, product name, etc. In other words, the type information is information indicating the component product number, product name, etc.

[0101] For example, if the types of the multiple components include the product names "Flooring Material A," "Flooring Material B," and "Flooring Material C," the multiple first learning images include an image that includes a "Flooring Material A" component but does not include a joint where two "Flooring Material A" components are joined (e.g., an image that includes one "Flooring Material A" component), an image that includes "Flooring Material B" but does not include a joint where two "Flooring Material B" components are joined (e.g., an image that includes one "Flooring Material B" component), and an image that includes "Flooring Material C" but does not include a joint where two "Flooring Material C" components are joined (e.g., an image that includes one "Flooring Material C" component). Also, for example, the learning type information includes information indicating the position (e.g., coordinate information, etc.) of "Flooring Material A" in an image that includes "Flooring Material A," information indicating the position of "Flooring Material B" in an image that includes "Flooring Material B," and information indicating the position (e.g., coordinate information, etc.) of "Flooring Material C" in an image that includes "Flooring Material C."

[0102] The identifying unit 144 may identify, as the type of member, for example, the material such as wood or resin, the shape such as square or rectangle, or the pattern such as wood grain or lattice pattern.

[0103] Furthermore, when components 201 and 202 are the same component (for example, when it is predetermined that the image acquired by image acquisition unit 110 contains only one type of component), identification unit 144 may identify the type of the first component contained in image 300 based on the identification result of the type of component contained in area 310 and the identification result of the type of component contained in area 314.

[0104] The identification unit 144 determines the type of the first member (combination determination) based on the classification scores obtained by inputting each of the plurality of partial images, and outputs type information as information indicating the determined type.

[0105] 4, for example, the identification unit 144 uses a first identification model to obtain an identification result for the type of component included in the region 310 and an identification result for the type of component included in the region 314. The first identification model outputs, for example, probability information (classification score) indicating which of the multiple components is most likely to be the component.

[0106] For example, if the types of components are "Flooring Material A," "Flooring Material B," and "Flooring Material C," the first identification model outputs classification scores indicating the types of components included in region 310, linked to information indicating the types of components, such as "Flooring Material A": 0.7, "Flooring Material B": 0.2, and "Flooring Material C": 0.1. Similarly, for example, the first identification model outputs classification scores indicating the types of components included in region 314, such as "Flooring Material A": 0.8, "Flooring Material B": 0.1, and "Flooring Material C": 0.1.

[0107] For example, the identification unit 144 identifies the type of component with the highest classification score among the classification scores obtained by inputting each of the multiple images in the image as the type of the first component, and outputs information indicating the type as type information. In the example described above, the identification unit 144 identifies the type of the first component as "Flooring Material A" by majority vote because the highest classification score indicating the type of component included in region 310 is "Flooring Material A" and the highest classification score indicating the type of component included in the partial image of region 314 is "Flooring Material A." For example, the identification unit 144 outputs information indicating "Flooring Material A" to the display unit 130 as type information. As a result, for example, the display unit 130 displays "Flooring Material A" as shown in FIG. 2.

[0108] Alternatively, the identification unit 144 may identify the type of the first component based on the average value of the classification scores obtained by inputting each of the multiple partial images, and output information indicating the type as type information. In the above example, for example, the identification unit 144 identifies the type of the first component as "Flooring Material A" because the average value of the classification scores for "Flooring Material A" is 0.75, the average value of the classification scores for "Flooring Material B" is 0.15, and the average value of the classification scores for "Flooring Material C" is 0.1.

[0109] Alternatively, the identification unit 144 may identify the type of the first component based on the multiplied value of the classification scores obtained by inputting each of the multiple partial images, and output information indicating the type as type information. In the above example, for example, the identification unit 144 identifies the type of the first component as "Flooring Material A" because the multiplied value of the classification scores for "Flooring Material A" is 0.56, the multiplied value of the classification scores for "Flooring Material B" is 0.02, and the multiplied value of the classification scores for "Flooring Material C" is 0.01.

[0110] The identification unit 144 may output information (for example, classification score) output from the first identification model as type information as is.

[0111] In addition, the first identification model may determine the type of the first component based on the classification scores obtained by inputting each of multiple regions in the image, as in the processing of the identification unit 144 described above, and output type information as information indicating the determined type.

[0112] The discrimination models (the first discrimination model and the second discrimination model described above) are, for example, machine learning models using neural networks such as deep learning (e.g., convolutional neural networks (CNNs)), but may also be other machine learning models.

[0113] The identification model is stored in advance in the storage unit 150, for example.

[0114] Note that storing an identification model means storing information such as network parameters in the identification model, a calculation algorithm (machine learning algorithm), etc. The algorithm may be any algorithm.

[0115] The storage unit 150 is a storage device that stores programs executed by the information processing unit 140 to perform information processing, and information necessary for the information processing. The storage unit 150 is realized by, for example, an HDD (Hard Disk Drive), a semiconductor memory, etc. Information stored in the storage unit 150 includes, for example, control programs executed by each processing unit, a first identification model, a second identification model, an image, information on the number of regions, etc.

[0116] The image identification device 10 may further include a condition acquisition unit that acquires the shooting conditions when an image is acquired by the image acquisition unit 110. The shooting conditions include, for example, the distance from the image acquisition unit 110 to the above-mentioned first member that is the subject, the attitude of the image acquisition unit 110 (image identification device 10), the zoom setting of the image acquisition unit 110, and the brightness around the subject. Specifically, the attitude of the image acquisition unit 110 includes the respective angles of pan, tilt, and roll. The shooting conditions acquired by the condition acquisition unit are stored in the storage unit 150 in association with the image. Specifically, the condition acquisition unit is realized by various sensors such as a distance sensor such as a TOF (Time Of Flight) sensor, an acceleration sensor, a gyro sensor, and a brightness sensor.

[0117] The image identification device 10 may further include a position acquisition unit that acquires position information of the image acquisition unit 110. The position acquired by the position acquisition unit is a three-dimensional position, but may be a two-dimensional position. The position information also includes attitude information that indicates the attitude (orientation) of the image acquisition unit 110. The position acquisition unit is realized by an acceleration sensor, a gyro sensor, a GPS (Global Positioning System) module, etc.

[0118] In machine learning performed in advance on the discriminative model, multiple images with different shooting conditions such as distance, camera orientation, color temperature of the lighting during shooting, positional deviation, etc. may be used as training data. This information may be input together with the image as input to the discriminative model.

[0119] [Operation] Next, the operation of the image identification device 10 will be described.

[0120] FIG. 7 is a flowchart illustrating the operation procedure of the image identification device 10 according to the embodiment.

[0121] First, the image acquisition unit 110 acquires an image (S101). For example, the image acquisition unit 110 acquires (generates) an image including a first member, which is an interior material, by photographing a subject such as an interior material installed in a building. The image acquisition unit 110 acquires an image when, for example, a photographing operation is accepted by the operation acceptance unit 120. The image acquired by the image acquisition unit 110 is stored in, for example, the storage unit 150.

[0122] Next, the setting unit 141 sets one or more regions in the image acquired by the image acquiring unit 110 (S102). In this example, for example, the setting unit 141 sets one region in the image acquired by the image acquiring unit 110.

[0123] Next, the first determination unit 142 determines whether or not a seam is included in the area set by the setting unit 141 (that is, the partial image of the area) in the image acquired by the image acquisition unit 110 (S103).

[0124] If the first determination unit 142 determines that the region includes a seam (Yes in S103), the process returns to step S102, and the setting unit 141 resets the region in the image acquired by the image acquisition unit 110.

[0125] On the other hand, if the first determination unit 142 determines that the region does not include a seam (No in S103), the second determination unit 143 determines whether or not the number of regions that do not include a seam is equal to or greater than a predetermined number (S104). That is, the second determination unit 143 determines whether or not the number of partial images that do not include a seam is equal to or greater than a predetermined number.

[0126] If the second determination unit 143 determines that the number of regions that do not include a seam is less than the predetermined number (No in S104), the process returns to step S102, and the setting unit 141 sets further regions in the image acquired by the image acquisition unit 110. In other words, the image identification device 10 continues setting regions until the number of regions that do not include a seam reaches the predetermined number.

[0127] On the other hand, if the second determination unit 143 determines that the number of regions not including a seam is equal to or greater than the predetermined number (Yes in S104), the identification unit 144 uses the first identification model to identify the type of the first member included in the image and outputs type information indicating the type of the first member (S105). As a result, the image shown in FIG. 2 is displayed on the display unit 130.

[0128] [Effects, etc.] As described above, the image identification device 10 according to the embodiment includes an image acquisition unit 110 that acquires an image including a first member, a setting unit 141 that sets one or more regions in the image (for example, regions 310, 311 shown in FIG. 6), a first determination unit 142 that determines whether or not each of the one or more regions in the image includes a seam between two first members, and an identification unit 144 that uses a first identification model constructed based on first learning images of multiple members of different types including the first member that do not include a seam between two members to identify the type of the first member by inputting a region in the one or more regions in the image that the first determination unit 142 determines does not include a seam between the two first members, and outputs type information indicating the type of the first member.

[0129] If an image contains a seam, the seam may cause the identification unit 144 to make an erroneous identification. Therefore, the image identification device 10 uses an image (partial image) that does not contain a seam to identify the type of component. This improves the accuracy of identifying the type of component. Furthermore, the user does not need to be conscious of the position at which the image is taken so as not to include a seam when photographing the component, and the identification accuracy is improved. In other words, the identification accuracy is improved regardless of the position at which the user photographs the component, which prevents the image identification device 10 from becoming difficult to use for the user.

[0130] Furthermore, constructing a classification model that can accurately classify images containing seams may require a huge amount of data. Therefore, when the image classification device 10 is used as a terminal device such as a tablet device, the data volume becomes huge, making it difficult to use the classification model. Therefore, the image classification device 10 is configured to use images that do not contain seams, thereby improving classification accuracy without requiring a huge amount of data for the classification model.

[0131] Furthermore, for example, the image identification device 10 further includes a second determination unit 143 that determines whether the number of regions that the first determination unit 142 has determined not to include a seam between the two first members among the one or more regions is equal to or greater than a predetermined number. In this case, for example, if the second determination unit 143 determines that the number is less than the predetermined number, the setting unit 141 further sets a region that is at least partially different from each of the one or more regions. In this case, for example, the first determination unit 142 further determines whether the region further set by the setting unit 141 includes a seam between the two first members.

[0132] According to this, a predetermined number or more of images (partial images) are used to identify the type of component, thereby further improving the identification accuracy.

[0133] Furthermore, for example, the setting unit 141 further resets an area that is at least partially different from the area that the first determination unit 142 determined to include the seam between the two first members. In this case, for example, the first determination unit 142 further determines whether or not the area reset by the setting unit 141 includes the seam between the two first members.

[0134] This makes it easier to reset the area that does not include seams.

[0135] Furthermore, for example, in the above resetting, the setting unit 141 resets the region based on the position of the seam between the two first members.

[0136] According to this, for example, by setting the area so as not to include seams, it is possible to prevent the process of setting the area from being repeated many times.

[0137] Also, for example, in the above resetting, the setting unit 141 resets the area by moving the area that the first determination unit 142 has determined to include a seam between two first members in a direction intersecting the longitudinal direction of the seam.

[0138] For example, if the component is an interior material, the seam is often a straight line. Therefore, by moving the area determined to include the seam in a direction intersecting the longitudinal direction of the seam and resetting the area, it becomes easier to set the area so that it does not include the seam.

[0139] Furthermore, for example, the first determination unit 142 detects straight lines included in the image to determine whether or not a seam between two first members is included.

[0140] As described above, for example, when the component is an interior material, the seam is often a straight line, so this method makes it possible to accurately detect the seam in the image.

[0141] Furthermore, for example, the first judgment unit 142 uses a second identification model constructed based on a plurality of first learning images and a plurality of second learning images including a seam between two components of a plurality of different types including the first component, to input the one or more regions in the image and output seam information indicating whether or not each of the one or more regions includes a seam between the two first components, thereby determining whether or not each of the one or more regions includes a seam between the two first components.

[0142] This allows seams in the image to be detected with even greater accuracy.

[0143] Furthermore, for example, each of the multiple components is an interior material to be applied to a building.

[0144] As described above, for example, when components are interior materials, etc., the joints are often straight lines. Therefore, for example, when components (joints) are determined using straight lines included in an image, the components can be identified with particularly high accuracy as interior materials to be installed in a building.

[0145] Furthermore, for example, the image identification device 10 further includes a display unit 130 that displays an image indicating the type of the first member indicated by the type information output by the identification unit 144.

[0146] This allows, for example, a user of the image identification device 10 to easily recognize the type of the first member by checking the display unit 130.

[0147] In addition, an image identification method according to an embodiment executed by a computer such as the image identification device 10 acquires an image including a first member (S101), sets one or more regions in the image (S102), and determines whether each of the one or more regions in the image includes a seam between two first members (S103). Using a first identification model constructed based on first learning images of a plurality of different types of members including the first member that do not include a seam between the two members, an area among the one or more regions in the image that is determined not to include a seam between the two first members is used as input to identify the type of the first member, and type information indicating the type of the first member is output (S105).

[0148] This provides the same effects as the image recognition device 10 according to one aspect of the present disclosure.

[0149] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0150] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0151] For example, the image identification device 10 does not need to include the operation reception unit 120.

[0152] Also, for example, the image identification device 10 may not include the second determination unit 143. In this case, for example, after executing step S103, the image identification device 10 may execute step S105 using the area that does not include a seam.

[0153] Furthermore, for example, the image recognition device 10 may further include a learning unit (processing unit) for learning the recognition model. In this case, the storage unit 150 may store the above-mentioned training data.

[0154] In the above embodiment, the image identification device is realized as a single device, but it may be realized by multiple devices. When the image identification device is realized by multiple devices, the components of the image identification device described in the above embodiment may be distributed among the multiple devices in any manner.

[0155] For example, the image recognition device may be realized as a client-server system. In this case, the client device is a mobile terminal that acquires images, accepts user operations, and displays determination results, while the server device is an information terminal that performs information processing using images, such as matching features. The image recognition device may also be realized as a robot-type terminal that moves within a building.

[0156] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0157] In the above-described embodiments, each component (each processing unit) may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU (Central Processing Unit) or a processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0158] Furthermore, each component may be realized by hardware. Each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0159] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0160] For example, the present disclosure may be realized as an image recognition method executed by a computer such as an image recognition device. Furthermore, the present disclosure may be realized as a program for causing a computer to execute the image recognition method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0161] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0162] The present disclosure is useful as an image identification device, an image identification method, a program, and the like that can accurately identify the type of component included in an image. [Explanation of symbols]

[0163] 10 Image recognition device 110 Image acquisition unit 120 Operation reception section 130 Display section 140 Information Processing Department 141 Setting section 142 1st Judgment Department 143 Second Judgment Section 144 Identification unit 150 Storage section 200, 201, 202 parts 210, 211 seams 300 images Areas 310, 311, 312, 313, 314, and 315

Claims

1. an image acquisition unit that acquires an image including the first member; a setting unit that sets one or more regions in the image; a first determination unit that determines whether or not a seam between two of the first members is included in each of the one or more regions in the image; an identification unit that uses a first identification model constructed based on a first learning image of each of a plurality of different types of members including the first member that does not include a seam between two of the members, and that receives as input an area determined by the first determination unit not to include a seam between two of the first members among the one or more areas in the image, to identify the type of the first member, and outputs type information indicating the type of the first member. Image recognition device.

2. The method further includes a second determination unit that determines whether or not the number of regions that the first determination unit has determined not to include a seam between two of the first members is equal to or greater than a predetermined number, When the second determination unit determines that the number is less than the predetermined number, the setting unit further sets an area at least partially different from each of the one or more areas, The first determination unit further determines whether or not the region further set by the setting unit includes a seam between two of the first members. The image recognition device according to claim 1 .

3. The setting unit further resets a region that is at least partially different from the region determined by the first determination unit to include a seam between the two first members, The first determination unit further determines whether or not a seam between two of the first members is included in the region reset by the setting unit. The image recognition device according to claim 1 or 2.

4. In the resetting, the setting unit resets the region based on a position of a seam between the two first members. The image recognition device according to claim 3 .

5. In the resetting, the setting unit resets the area by moving the area that the first determination unit determined to include the seam between the two first members in a direction intersecting with the longitudinal direction of the seam. The image recognition device according to claim 3 .

6. The first determination unit determines whether or not a seam between two of the first members is included by detecting a straight line included in the image. The image recognition device according to claim 1 .

7. The first determination unit uses a second discriminative model constructed based on the plurality of first learning images and second learning images including a seam between two of a plurality of mutually different types of members including the first member, to input the one or more regions in the image and output seam information indicating whether or not a seam between the two first members is included in each of the one or more regions, thereby determining whether or not a seam between the two first members is included in each of the one or more regions. The image recognition device according to claim 1 .

8. Each of the plurality of components is an interior material to be applied to a building. The image recognition device according to claim 1 .

9. The apparatus further includes a display unit that displays an image indicating the type of the first member indicated by the type information output by the identification unit. The image recognition device according to claim 1 .

10. An image identification method performed by an image identification device having an image acquisition unit and an information processing unit, The image acquisition unit acquiring an image including the first member; The information processing unit defining one or more regions in the image; determining whether each of the one or more regions in the image includes a seam between two of the first members; Using a first identification model constructed based on a first learning image that does not include a seam between two of a plurality of mutually different types of members including the first member, the type of the first member is identified using, as input, a region determined not to include a seam between the two first members among the one or more regions in the image, and type information indicating the type of the first member is output. Image identification methods.

11. A program for causing a computer to execute the image identification method according to claim 10.

Images