MODEL GENERATION SYSTEM, SHAPE RECOGNITION SYSTEM, MODEL GENERATION METHOD, SHAPE RECOGNITION METHOD, AND COMPUTER PROGRAM

Abstract

A model generation system (1) is provided with: an extraction means (110) that extracts an object area part from a target image, the object area part being the area occupied by an object; and a generation means (120) that executes machine learning using the object area part as an input, and generates a shape classification model (300) for classifying the shape of the object. The shape classification model generated in this manner can be used to enable appropriate recognition of the shape of an object in an image.

Description

[Technical field]

[0001] The present invention relates to the technical fields of a model generation system, a shape recognition system, a model generation method, a shape recognition method, and a computer program for recognizing the shape of an object. [Background technology]

[0002] Known systems of this type recognize objects in images. For example, Patent Document 1 discloses a technique for identifying objects using the features of the objects (texture, color, shape, boundary, etc.). As other related techniques, Patent Document 2 discloses a technique for inferring that objects are the same from their shapes. Patent Document 3 discloses a technique for searching for images using the similarity of objects in the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-507855 [Patent Document 2] JP 2019-070467 A [Patent Document 3] Japanese Patent Application Publication No. 10-240771 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to recognize the shape of an object, a method of performing machine learning using information about the shape is conceivable. However, with the technology described in the above-mentioned Patent Document 1, it is extremely difficult to capture and learn only the shape feature from among various features included in the image, such as differences in the background and differences in the color of the object. In other words, even if the above-mentioned technology is applied, it is not easy to build a system that appropriately recognizes the shape of an object.

[0005] The present invention has been made in consideration of the above problems, and has an objective of providing a model generation system, a shape recognition system, a model generation method, a shape recognition method, and a computer program capable of appropriately recognizing the shape of an object. [Means for solving the problem]

[0006] One aspect of the model generation system of the present invention includes an extraction means for extracting an object area portion, which is an area occupied by an object, from a target image, and a generation means for performing machine learning using the object area portion as an input to generate a shape classification model for classifying the shape of the object.

[0007] One aspect of the shape recognition system of the present invention comprises an extraction means for extracting an object area portion, which is an area occupied by an object, from a target image, and an estimation means for estimating the shape of the object in the object area portion using a shape classification model for classifying the shape of the object.

[0008] One aspect of the model generation method of the present invention extracts an object region, which is the area occupied by an object, from a target image, and performs machine learning using the object region as input to generate a shape classification model for classifying the shape of the object.

[0009] One aspect of the shape recognition method of the present invention extracts an object area portion, which is the area occupied by an object, from a target image, and estimates the shape of the object in the object area portion using a shape classification model for classifying the shape of the object.

[0010] One aspect of the computer program of the present invention operates a computer to extract an object area portion, which is an area occupied by an object, from a target image, perform machine learning using the object area portion as an input, and generate a shape classification model for classifying the shape of the object.

[0011] One aspect of the computer program of the present invention operates a computer to extract an object area portion, which is an area occupied by an object, from a target image, and estimate the shape of the object in the object area portion using a shape classification model for classifying the shape of the object. Effect of the Invention

[0012] According to one aspect of each of the above-described model generation system, shape recognition system, model generation method, shape recognition method, and computer program, it is possible to appropriately recognize the shape of an object. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a model generation system according to a first embodiment. [Diagram 2] 1 is a block diagram showing functional blocks of a model generation system according to a first embodiment. FIG. [Diagram 3] FIG. 1 is a conceptual diagram showing extraction of an object region using an instance segmentation model. [Figure 4] 4 is a flowchart showing the flow of operations of the model generation system according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing functional blocks of a model generation system according to a second embodiment. [Figure 6] 10 is a flowchart showing the flow of operations of the model generation system according to the second embodiment. [Figure 7] FIG. 11 is a block diagram showing functional blocks of a shape recognition system according to a third embodiment. [Figure 8] 13 is a flowchart showing the flow of operations of the shape recognition system according to the third embodiment. [Figure 9] FIG. 13 is a conceptual diagram showing a specific operation example of the shape recognition system according to the third embodiment. [Figure 10] FIG. 13 is a diagram (part 1) showing a specific output example of the shape recognition system according to the third embodiment. [Figure 11] FIG. 13 is a diagram (part 2) showing a specific output example of the shape recognition system according to the third embodiment. [Figure 12] FIG. 13 is a block diagram showing functional blocks of a shape recognition system according to a fourth embodiment. [Figure 13] 13 is a flowchart showing the flow of operations of the shape recognition system according to the fourth embodiment. [Figure 14] 13 is a flowchart showing a flow of operations of a shape recognition system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of a model generation system, a shape recognition system, a model generation method, a shape recognition method, and a computer program will be described with reference to the drawings.

[0015] First Embodiment First, a model generation system according to the first embodiment will be described with reference to FIGS.

[0016] (Hardware configuration) The hardware configuration of the model generation system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the model generation system according to the first embodiment.

[0017] 1, a model generation system 10 according to the first embodiment includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The model generation system 10 may further include an input device 15 and an output device 16. The CPU 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16 are connected via a data bus 17.

[0018] The CPU 11 reads a computer program. For example, the CPU 11 is configured to read a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the CPU 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The CPU 11 may obtain (i.e., read) a computer program from a device (not shown) disposed outside the model generation system 10 via a network interface. The CPU 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the computer program that the CPU 11 reads. In particular, in this embodiment, when the CPU 11 executes the computer program that it reads, a functional block for generating a shape classification model that identifies the shape of an object is realized in the CPU 11.

[0019] The RAM 12 temporarily stores a computer program executed by the CPU 11. The RAM 12 temporarily stores data that is temporarily used by the CPU 11 when the CPU 11 is executing the computer program. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0020] The ROM 13 stores computer programs executed by the CPU 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a programmable ROM (P-ROM).

[0021] The storage device 14 stores data that is to be stored long-term by the model generation system 10. The storage device 14 may operate as a temporary storage device for the CPU 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0022] The input device 15 is a device that receives input instructions from a user of the model generation system 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel.

[0023] The output device 16 is a device that outputs information related to the model generation system 10 to the outside. For example, the output device 16 may be a display device (for example, a display) that can display information related to the model generation system 10.

[0024] (System Configuration) Next, the functional configuration of the model generation system 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing functional blocks included in the model generation system according to the first embodiment.

[0025] 2, the model generation system 10 according to the first embodiment includes an object region partial extraction unit 110 and a model generation unit 120. These functional blocks are realized, for example, in a CPU 11 (see FIG. 1).

[0026] The object region portion extraction unit 110 is configured to be able to extract an object region portion, which is a region occupied by an object of a predetermined shape (in other words, a shape to be recognized), from image data input to the system. The object region portion extraction unit 110 extracts the object region portion using an instance segmentation model 200. Here, a method of extracting an object region portion using the instance segmentation model 200 will be described with reference to Fig. 3. Fig. 3 is a conceptual diagram showing the extraction of an object region portion using an instance segmentation model.

[0027] As shown in Fig. 3, by using the instance segmentation model 200, it is possible to extract only the object region from an image containing an object. For example, from an image of a round object such as an apple or a golf ball, it is possible to extract a mask image in which only the area occupied by the object (i.e., only the round area) is cut out. Similarly, from an image of a square object such as a smartphone or a computer monitor, it is possible to extract a mask image in which only the area occupied by the object (i.e., only the square area) is cut out.

[0028] The instance segmentation model 200 is a model that extracts an object region by processing an image for each of a plurality of unit regions (for example, processing an image for each pixel), but since this technology is already known, a detailed description thereof will be omitted here. Also, although a method using an instance segmentation model is given here, the object region may be extracted by other methods.

[0029] Returning to Fig. 2, the object region portion extraction unit 110 outputs an object region portion extracted using the instance segmentation model 200. Information on the object region portion output from the object region portion extraction unit 110 is configured to be output to the model generation unit 120. Note that the object region portion extraction unit 110 is a specific example of "extraction means".

[0030] The model generation unit 120 is configured to be able to execute machine learning using the object region portion extracted by the object region portion extraction unit 110 as input data (in other words, teacher data). The model generation unit 120 generates a shape classification model for recognizing the shape of an object through this machine learning. Note that the object region portion may be manually annotated (for example, information indicating what the extracted shape actually is) before being input to the model generation unit 120. An existing learning method can be appropriately applied to the machine learning of the model generation unit 120. The model generation unit 120 is a specific example of a "generation means".

[0031] (Operation description) Next, the flow of operations of the model generation system 10 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of operations of the model generation system according to the first embodiment.

[0032] 4, an image data group consisting of a plurality of image data is first input to the model generation system 10 according to the first embodiment (step S101). The image data group input here is image data in which an object of a predetermined shape (for example, a round object or a square object) to be recognized by the shape classification model is captured. However, it is not necessary for all image data to include an object of the predetermined shape.

[0033] Next, the object region portion extraction unit 110 extracts an object region portion occupied by an object of a predetermined shape from the input image data group (step S102). Then, the model generation unit 120 executes machine learning using the extracted object region portion as input data (step S103). The model generation unit 120 outputs a shape classification model for recognizing the shape of the object as a result of the machine learning (step S104).

[0034] (Technical effect) Next, technical effects obtained by the model generation system 10 according to the first embodiment will be described.

[0035] As described in FIG. 1 to FIG. 4, in the model generation system 10 according to the first embodiment, an object region portion is extracted using the instance segmentation model 200, and a shape classification model is generated by machine learning using the object region portion as input. By using the shape classification model generated in this way, it is possible to appropriately recognize the shape of an object in an image. More specifically, by extracting the object region portion, it is possible to appropriately extract only information about the shape of an object contained in an image. For example, in a mask image as shown in FIG. 2, information other than the shape (for example, information about color and pattern, etc.) is scraped off, and only information about the shape of the object is reliably extracted. In addition, for objects that overlap each other in an image (i.e., objects whose shapes are difficult to distinguish due to overlapping), it is easy to distinguish what kind of shape they are by extracting only the object region portion. Therefore, according to the model generation system 10 according to the first embodiment, it is possible to generate a shape classification model that can appropriately recognize the shape of an object.

[0036] In particular, in this embodiment, by generating a shape classification model using the object region as an input, it is possible to realize recognition that allows for ambiguity in shapes. Specifically, it is possible to recognize ambiguous shapes such as round-like shapes and square-like shapes (i.e., shapes that are far from perfect squares or circles).

[0037] <Second embodiment> Next, a model generation system 10 according to a second embodiment will be described with reference to Fig. 5 and Fig. 6. The second embodiment differs from the first embodiment in some configurations and operations, and other parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit descriptions of other overlapping parts as appropriate.

[0038] (System Configuration) First, the functional configuration of the model generation system 10 according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing functional blocks included in the model generation device according to the second embodiment. In Fig. 5, the same components as those shown in Fig. 2 are denoted by the same reference numerals.

[0039] 5, the model generating device 10 according to the second embodiment includes an object region partial extraction unit 110, a model generating unit 120, a designated image extraction unit 130, and a box region extraction unit 140. That is, the model generating device 10 according to the second embodiment is configured to further include a designated image extraction unit 130 and a box region extraction unit 140 in addition to the configuration of the first embodiment (see FIG. 2).

[0040] The designated image extraction unit 130 is configured to be able to extract only images including an object of a predetermined shape to be recognized from among a group of image data (i.e., a plurality of image data) input to the model generation system 10. The designated image extraction unit 130 may be configured to be able to specify a predetermined shape. In this case, for example, when a user specifies a predetermined shape (may be a plurality of shapes), the designated image extraction unit 130 extracts only images including an object of the specified predetermined shape (hereinafter, appropriately referred to as "designated images"). More specifically, for example, when a user specifies a shape of "round", only images including round objects such as apples and balls are extracted from among a plurality of images. The designated image extraction unit 130 extracts the designated image using the instance segmentation model 200. However, the designated image extraction unit 130 may extract the designated image without using the instance segmentation model 200. The designated image extracted by the designated image extraction unit 130 is configured to be output to the box region extraction unit 140. The designated image extraction unit 130 is a specific example of the "third extraction means".

[0041] Box region extraction unit 140 is configured to be able to extract a box region (specifically, a rectangular region surrounding an object) indicating the position of an object in an image from a specified image extracted by specified image extraction unit 130 (i.e., an image including an object of a predetermined shape). Box region extraction unit 140 may extract multiple box regions from one specified image. Box region extraction unit 140 extracts the box region using instance segmentation model 200. However, box region extraction unit 140 may extract box region extraction unit 140 without using instance segmentation model 200. The box region extracted by box region extraction unit 140 is configured to be output to object region portion extraction unit 110. Box region extraction unit 140 is a specific example of a "second extraction means".

[0042] (Operation description) Next, the flow of operations of the model generation system 10 according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of operations of the model generation system according to the second embodiment. In Fig. 6, the same processes as those shown in Fig. 4 are denoted by the same reference numerals.

[0043] As shown in FIG. 6, when the model generating system 10 according to the second embodiment is operated, an image data group consisting of a plurality of image data is first input (step S101).

[0044] Next, the designated image extraction unit 130 extracts a designated image including an object of a predetermined shape from the input image data group (step S102). Then, the box region extraction unit 140 extracts a box region indicating the position of the object from the designated image (step S103).

[0045] Next, the object region portion extraction unit 110 extracts an object region portion occupied by an object of a predetermined shape from the extracted box region (step S102). Specifically, the object region portion extraction unit 110 extracts the object region portion by processing the rectangular region extracted as the box region, for example, on a pixel-by-pixel basis.

[0046] Next, the model generation unit 120 executes machine learning using the extracted object region portion as input data (step S103). The model generation unit 120 outputs a shape classification model for recognizing the shape of the object as a result of the machine learning (step S104).

[0047] (Technical effect) Next, technical effects obtained by the model generation system 10 according to the second embodiment will be described.

[0048] As described in Fig. 5 and Fig. 6, in the model generation system 10 according to the second embodiment, a specified image including an object of a predetermined shape is extracted from an image data group, and a box region indicating the position of the object is further extracted from the specified image. In this way, the object region portion can be extracted more easily and with high accuracy. As a result, the model generation system 10 according to the second embodiment can generate a shape classification model capable of more appropriately recognizing the shape of an object.

[0049] <Modification> In the above example, the instance segmentation model 200 is used to extract information about the shape of an object. However, information about the color information of an object may be extracted.

[0050] For example, by using the intensity segmentation model 200, color information (e.g., R, G, B information) of the object region can be extracted. Therefore, it is possible to assign color information of the object (e.g., red, green, blue, yellow, white, black, etc.) from the distribution of R, G, B on the object. In this case, if the color is almost uniformly the same on the object, it may be treated as one color, and if various colors are distributed, special color information such as "colorful" may be assigned. Alternatively, a pattern may be determined from the color distribution of the object, and information regarding the pattern of the object may be assigned.

[0051] The above-mentioned color information may be added to the information about the shape. In this case, the model generation unit 120 may learn information about the shape of the object and information about the color, and generate a model capable of recognizing the shape and color of the object. Alternatively, the color information may be added instead of the information about the shape. In this case, the model generation unit 120 may learn information about the color of the object, and generate a model capable of recognizing the color of the object.

[0052] <Third embodiment> Next, a shape recognition system 20 according to the third embodiment will be described with reference to Fig. 7 to Fig. 11. The shape recognition system 20 according to the third embodiment has some configuration and operation in common with the model generation system 10 according to the first and second embodiments described above (for example, the hardware configuration may be the same as that of the model generation system 10 shown in Fig. 1). Therefore, in the following, the description of matters that have already been described will be omitted, and non-overlapping parts will be described in detail.

[0053] (System Configuration) First, the functional configuration of the shape recognition system 20 according to the third embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing functional blocks of the shape recognition system according to the third embodiment. In Fig. 7, the same components as those shown in Figs. 2 and 5 are denoted by the same reference numerals.

[0054] 7, the shape recognition system 20 according to the third embodiment includes an object region portion extraction unit 110 and a shape estimation unit 150. Note that the object region portion extraction unit 110 is similar to that included in the model generation system 10 according to the first and second embodiments (see FIGS. 2 and 5), and is configured to be able to extract an object region portion from image data by using an instance segmentation model 200.

[0055] Shape estimation unit 150 is configured to be able to estimate the shape of an object from the object region portion extracted by object region portion extraction unit 110. Shape estimation unit 150 estimates the shape of the object using a shape classification model 300 (i.e., a model generated by model generation system 10 according to the first and second embodiments). Shape estimation unit 150 is a specific example of an "estimation means".

[0056] (Operation description) Next, the flow of operations of the shape recognition system 20 according to the third embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of operations of the shape recognition system 20 according to the third embodiment.

[0057] As shown in Fig. 8, image data is first input to the shape recognition system 20 according to the third embodiment (step S301). The image input here is an image including an object whose shape is to be recognized. Note that multiple images may be input. In this case, the following process may be performed for each image.

[0058] Next, the object region portion extraction unit 110 extracts an object region portion occupied by an object of a predetermined shape from the input image (step S302). Then, the shape estimation unit 150 estimates the shape of the object corresponding to the extracted object region portion using the shape classification model 300 (step S303). Finally, the shape estimation unit 150 outputs information indicating the shape of the object as an estimation result (step S304).

[0059] The shape estimation unit 150 may output information indicating which of the predetermined shapes the object corresponding to the object region portion has (e.g., whether it is round or square). Specifically, it may output a score indicating the roundness or squareness of the object. This score may be output, for example, as a numerical value indicating the likelihood that the object is a round object (or a square object). Furthermore, if the object has a shape that cannot be classified into any of the predetermined shapes, information such as "unestimateable" may be output.

[0060] <Specific output example> Next, a specific example of output from the shape recognition system 20 according to the third embodiment will be described with reference to Figs. 9 to 11. Fig. 9 is a conceptual diagram showing a specific example of operation of the shape recognition system according to the third embodiment. Fig. 10 is a diagram (part 1) showing a specific example of output from the shape recognition system according to the third embodiment. Fig. 11 is a diagram (part 2) showing a specific example of output from the shape recognition system according to the third embodiment.

[0061] 9 includes a keyboard and a mouse. When the instance segmentation model 200 is applied to such an image, the object region portions of the keyboard and the mouse can be extracted.

[0062] Next, when shape classification model 300 is applied to the object region portion, a score (0 to 1) indicating the shape of the object corresponding to the object region portion is displayed. Here, a score of "square (1.00)" is displayed for the keyboard. This result means that the keyboard in the image is very close to a square shape. On the other hand, a score of "circle (1.00)" is displayed for the mouse. This result means that the mouse in the image is very close to a round shape.

[0063] The image shown in FIG. 10 includes a refrigerator and a microwave oven. When shape recognition is performed on such an image in a similar manner, a score of "square (1.00)" is shown for the refrigerator. This result means that the refrigerator in the image is very close to a square shape. On the other hand, a score of "square (1.00)" is also shown for the microwave. This result means that the microwave in the image is very close to a square shape.

[0064] The image shown in FIG. 11 includes a monitor (TV), a keyboard, a mouse, and a cup. When shape recognition is performed on such an image in the same manner, a score of "square (1.00)" is shown for the monitor (tv). This result means that the monitor in the image is very close to a square shape. A score of "square (1.00)" is also shown for the keyboard. This result means that the keyboard in the image is very close to a square shape. A score of "circle (1.00)" is also shown for the mouse. This result means that the mouse in the image is very close to a round shape. Furthermore, a score of "circle (0.56)" is also shown for the cup. This result means that the cup in the image is very close to a round shape.

[0065] As described above, by displaying a score indicating the shape of an object, it is possible to intuitively grasp the shape of the object. In addition, depending on the size of the score, it is possible to determine how close the shape is to a round shape or how close the shape is to a square shape. Therefore, even if an object is not completely round, it can be determined as being somewhat roundish, and even if an object is not completely square, it can be determined as being somewhat squareish.

[0066] In the above example, the case where an object is recognized as being round or square has been described, but the object may be configured to be able to recognize shapes other than round and square. For example, the object may be configured to be able to recognize a triangle, a star, or even a more complicated shape.

[0067] (Technical effect) Next, technical effects obtained by the shape recognition system 20 according to the third embodiment will be described.

[0068] As described in FIG. 7 to FIG. 11, in the shape recognition system 20 according to the third embodiment, an object region portion is extracted using the instance segmentation model 200. Then, the shape of the object is estimated by using the shape classification model 300 for the object region portion. Here, as already described in the first and second embodiments, the shape classification model 300 is generated as a model capable of appropriately recognizing the shape of the object. In addition, since the object region portion is extracted using the instance segmentation model 200 and then shape estimation is performed, it is possible to estimate the shape of the object with extremely high accuracy.

[0069] In particular, in this embodiment, by using a shape classification model generated by inputting the object region portion, it is possible to realize recognition that allows for ambiguity in shape. Specifically, it is possible to recognize ambiguous shapes such as round-like shapes and square-like shapes (i.e., shapes that are far from perfect squares or circles).

[0070] <Fourth embodiment> Next, a shape recognition system 20 according to a fourth embodiment will be described with reference to Fig. 12 to Fig. 14. The fourth embodiment differs from the third embodiment in some configurations and operations, and other parts are generally similar. Therefore, the following will describe in detail the parts that differ from the third embodiment, and will omit descriptions of other overlapping parts as appropriate.

[0071] (System Configuration) First, the functional configuration of the shape recognition system 20 according to the fourth embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing functional blocks of the shape recognition system according to the fourth embodiment. In Fig. 12, the same components as those shown in Fig. 7 are denoted by the same reference numerals.

[0072] As shown in Fig. 12, shape recognition system 20 according to the fourth embodiment includes object region partial extraction unit 110, box region extraction unit 140, and shape estimation unit 150. That is, model generation device 10 according to the fourth embodiment is configured to further include box region extraction unit 140 in addition to the configuration of the third embodiment (see Fig. 7). Note that box region extraction unit 140 extracts a box region indicating the position of an object from an image, as described in the second embodiment.

[0073] (Operation description) Next, the flow of operations of the shape recognition system 20 according to the fourth embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the flow of operations of the shape recognition system according to the fourth embodiment. In Fig. 13, the same processes as those shown in Fig. 8 are denoted by the same reference numerals.

[0074] As shown in FIG. 13, when the shape recognition system 20 according to the fourth embodiment operates, image data is first input (step S301).

[0075] Next, box region extraction unit 140 extracts a box region indicating the position of an object from the input image (step S401). Then, object region portion extraction unit 110 extracts an object region portion occupied by an object of a predetermined shape from the extracted box region (step S302).

[0076] Next, shape estimation unit 150 estimates the shape of the object corresponding to the extracted object region portion using shape classification model 300 (step S303). Finally, shape estimation unit 150 outputs information indicating the shape of the object as an estimation result (step S304).

[0077] (Technical effect) Next, technical effects obtained by the shape recognition system 20 according to the fourth embodiment will be described.

[0078] 12 and 13, in the shape recognition system 20 according to the fourth embodiment, a box region indicating the position of an object is extracted from an input image. In this way, the object region can be extracted more easily and with higher accuracy. As a result, the shape recognition system 20 according to the fourth embodiment can estimate the shape of an object with higher accuracy.

[0079] <Modification> Next, a modified example of the shape recognition system 20 according to the above-mentioned fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of operations of the shape recognition system according to the modified example. In Fig. 14, the same processes as those shown in Fig. 13 are denoted by the same reference numerals.

[0080] In the fourth embodiment, an example of estimating the shape of an object included in image data has been given, but it is also possible to estimate the shape of an object included in video data using a similar method. In this case, the video data may be treated as a time-series set of multiple image data.

[0081] 14, when the shape recognition system 20 according to the modified example operates, first, N, which is a parameter for counting the number of repetitive processes, is set to "1" (step S501). Note that "1" here is a predetermined initial value, and the process of step S501 is a process for initializing N.

[0082] Next, video data is input to shape recognition system 20 (step S502). The video data is made up of T pieces of time-series image data. Shape recognition system 20 extracts the Nth image data from the video data (step S503).

[0083] Next, box region extraction unit 140 extracts a box region indicating the position of an object from the extracted Nth image (step S401). Then, object region portion extraction unit 110 extracts an object region portion occupied by an object of a predetermined shape from the extracted box region (step S302).

[0084] Next, shape estimation unit 150 estimates the shape of the object corresponding to the extracted object region portion using shape classification model 300 (step S303). Then, shape estimation unit 150 outputs information indicating the shape of the object as an estimation result (step S304).

[0085] Next, the shape recognition system 20 increments N (step S504). Then, the shape recognition system 20 determines whether N=T (step S505). In other words, it determines whether the processing of the last image data included in the video data has been completed.

[0086] If it is determined that N=T is not true (step S505: NO), the process is executed again from step S503. Thus, steps S503 to S504 are repeatedly executed until the process of the last image data included in the video data is completed. On the other hand, if it is determined that N=T is true (step S505: YES), the process ends.

[0087] According to the modified example described above, it is possible to appropriately recognize the shape of an object included in video data. With the spread of life logs and the like, video data is expected to be used in video search systems. When attempting to realize video search using free text queries, it will be necessary to respond to queries such as "When," "Where," "How," and "What."

[0088] Here, "when" queries can be answered using information obtained from the video's timestamp. "where" queries can be answered using the video's GPS information (latitude and longitude information). "what" queries can be answered using information that can be obtained using existing object detection. On the other hand, "how" queries are difficult to answer using information that can be obtained using existing technology.

[0089] In contrast, according to the shape recognition system 20 according to the above-described modified example, it is possible to respond to a "how" query using information related to the shape of an object recognized from video data. Specifically, a designation of the shape of an object is received from a user, and images including an object of the designated shape are searched for and output from among a plurality of image data constituting the video data. In this case, the designation of the shape by the user may be performed, for example, using the input device 15 (see FIG. 1). Furthermore, the output of the searched image may be performed, for example, using the output device 16 (see FIG. 1). In this way, for example, a search query such as "a round car I saw in Kyoto last August" can be responded to by extracting an object having a "round" shape. In this way, it is considered that the shape recognition system 20 according to the modified example has an extremely beneficial effect in a free text query search of video data.

[0090] <Additional Notes> The following supplementary notes are further disclosed regarding the above-described embodiment.

[0091] (Appendix 1) The model generation system described in Appendix 1 is a model generation system characterized by comprising an extraction means for extracting an object area portion, which is an area occupied by an object, from a target image, and a generation means for performing machine learning using the object area portion as an input to generate a shape classification model for classifying the shape of the object.

[0092] (Appendix 2) The model generation system described in Supplementary Note 2 is the model generation system described in Supplementary Note 1, characterized in that the extraction means processes the target image for each of a plurality of unit regions to extract the object region portion.

[0093] (Appendix 3) The model generation system described in Appendix 3 is the model generation system described in Appendix 1 or 2, further comprising a second extraction means for extracting a rectangular area containing the object from the target image, the extraction means being characterized in that it extracts the object area portion from the rectangular area.

[0094] (Appendix 4) The model generation system described in Appendix 4 is the model generation system described in any one of Appendixes 1 to 3, further comprising a designation means for designating a shape to be classified by the shape classification model, and a third extraction means for extracting an image including an object of the shape designated by the designation means as the target image from a plurality of images.

[0095] (Appendix 5) The model generation system described in Appendix 5 is the model generation system described in any one of Appendixes 1 to 4, further comprising a color information assignment means for detecting the color of the object area portion and assigning color information to the object area portion.

[0096] (Appendix 6) The shape recognition system described in Appendix 6 is a shape recognition system characterized by comprising an extraction means for extracting an object area portion, which is an area occupied by an object, from a target image, and an estimation means for estimating the shape of the object in the object area portion using a shape classification model for classifying the shape of the object.

[0097] (Appendix 7) The shape recognition system described in Supplementary Note 7 is the shape recognition system described in Supplementary Note 6, characterized in that the extraction means processes the target image for each of a plurality of unit regions to extract the object region portion.

[0098] (Appendix 8) The shape recognition system described in Appendix 8 is the shape recognition system described in Appendix 6 or 7, further comprising a second extraction means for extracting a rectangular area containing the object from the target image, the extraction means extracting the object area portion from the rectangular area.

[0099] (Appendix 9) The shape recognition system described in Appendix 9 is the shape recognition system described in any one of Appendixes 6 to 8, further comprising a receiving means for receiving a designation of the shape of the object, and an output means for outputting an image including an object of the designated shape from the multiple target images based on the estimation result of the estimation means.

[0100] (Appendix 10) The shape recognition system described in Appendix 10 is the shape recognition system described in any one of Appendixes 6 to 9, characterized in that the estimation means estimates the color of the object in the object area portion in addition to the shape of the object in the object area portion.

[0101] (Appendix 11) The model generation method described in Appendix 11 is a model generation method characterized by extracting an object region portion, which is an area occupied by an object, from a target image, performing machine learning using the object region portion as an input, and generating a shape classification model for classifying the shape of the object.

[0102] (Appendix 12) The shape recognition method described in Appendix 12 is a shape recognition method characterized by extracting an object area portion, which is an area occupied by an object, from a target image, and estimating the shape of the object in the object area portion using a shape classification model for classifying the shape of the object.

[0103] (Appendix 13) The computer program described in Appendix 13 is a computer program characterized by operating a computer to extract an object area portion, which is an area occupied by an object, from a target image, execute machine learning using the object area portion as an input, and generate a shape classification model for classifying the shape of the object.

[0104] (Appendix 14) The computer program described in Appendix 14 is a computer program characterized by operating a computer to extract an object area portion, which is an area occupied by an object, from a target image, and estimate a shape of the object in the object area portion using a shape classification model for classifying the shape of the object.

[0105] The present invention can be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and the model generation system, shape recognition system, model generation method, shape recognition method, and computer program that involve such modifications are also included in the technical idea of ​​the present invention. [Explanation of symbols]

[0106] 10 Model Generation System 20 Shape Recognition System 110 Object region partial extraction part 120 Model Generation Unit 130 Specified image extraction unit 140 Box area extraction part 150 Shape estimation part 200 instance segmentation model 300 Shape Classification Models

Claims

1. An extraction means for extracting an object area portion, which is an area occupied by an object, from a target image; a generating means for executing machine learning using the object region portion as training data to generate a shape classification model for discriminating the shape of the object; Equipped with The shape classification model outputs information indicating the shape of the object when the object region portion is input. A model generation system comprising:

2. 2. The model generating system according to claim 1, wherein said extracting means processes the target image for each of a plurality of unit regions to extract the object region portion.

3. a second extraction means for extracting a rectangular area including the object from the target image; The extraction means extracts the object region portion from the rectangular region.

3. The model generation system according to claim 1 or 2.

4. A designation means for designating a shape to be classified by the shape classification model; a third extraction means for extracting an image including an object having a shape designated by the designation means as the target image from among a plurality of images; 4. The model generation system according to claim 1, further comprising:

5. 5. The model generating system according to claim 1, further comprising a color information adding means for detecting a color of the object region portion and adding color information to the object region portion.

6. An extraction means for extracting an object area portion, which is an area occupied by an object, from a target image; an estimation means for estimating a shape of the object in the object region portion by using a shape classification model for discriminating the shape of the object; Equipped with The shape classification model outputs information indicating the shape of the object when the object region portion is input. A shape recognition system comprising:

7. Extracting an object region portion, which is a region occupied by an object, from the target image; performing machine learning using the object region portion as training data to generate a shape classification model for discriminating the shape of the object; The shape classification model outputs information indicating the shape of the object when the object region portion is input. A model generation method comprising:

8. Extracting an object region portion, which is a region occupied by an object, from the target image; estimating a shape of the object in the object region using a shape classification model for discriminating the shape of the object; The shape classification model outputs information indicating the shape of the object when the object region portion is input. A shape recognition method comprising:

9. Extracting an object region portion, which is a region occupied by an object, from the target image; performing machine learning using the object region portion as training data to generate a shape classification model for discriminating the shape of the object; The shape classification model outputs information indicating the shape of the object when the object region portion is input. A computer program for causing a computer to operate in such a manner.

10. Extracting an object region portion, which is a region occupied by an object, from the target image; estimating a shape of the object in the object region using a shape classification model for discriminating the shape of the object; The shape classification model outputs information indicating the shape of the object when the object region portion is input. A computer program for causing a computer to operate in such a manner.

Images