Object recognition system, object recognition method, and control program

Abstract

To provide an object recognition system, an object recognition method, and a control program capable of recognizing an object more accurately.SOLUTION: An object recognition system includes: acquisition means which acquires an input image obtained by imaging an object; integration means which inputs the input image to a plurality of object recognizers corresponding to multiple groups, respectively, to integrate output data output from the object recognizers; and output means which outputs information related to the integrated output data. Each of the groups is categorized based on intermediate features, which are output from an intermediate layer of a learning model by inputting a plurality of second learning images each including an object to the learning model, which is a base model, or a model trained using a first learning image including the object and ground-truth data corresponding to the object.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an object recognition system, an object recognition method, and a control program. [Background technology]

[0002] Techniques for recognizing objects from images have been studied. For example, techniques for recognizing buildings as objects from images captured from the air, such as aerial photographs or satellite images, have been studied for the purposes of interpreting building changes, creating maps, and creating city models.

[0003] For example, Patent Document 1 discloses a building extraction system in which building detectors for each of a plurality of area ranges are used to detect building regions from data to be processed. That is, in the building extraction system described in Patent Document 1, buildings in a training image are classified by area, and a building detector is trained for each group of classified buildings, thereby preparing a building detector suited to the area characteristics of the building. Then, building regions are detected by inputting the image, which is data to be processed, into the building detectors for the plurality of groups and integrating the outputs obtained. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-175139 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in addition to the size of the area, various other features such as shape and color may be combined to detect buildings, and arbitrarily defining the features used for classification can result in variations in the characteristics of building groups. This also applies to detected objects other than buildings. Therefore, there was room for improvement in the object recognition systems of the prior art to prevent arbitrary classification in order to recognize objects with higher accuracy.

[0006] An object of the present invention is to provide an object recognition system, an object recognition method, and a control program that are capable of recognizing objects with higher accuracy. [Means for solving the problem]

[0007] The object recognition system of the present invention comprises an acquisition means for acquiring an input image of an object, an integration means for inputting the input image to a plurality of object recognizers corresponding to each of a plurality of groups and integrating the output data output from each of the plurality of object recognizers, and an output means for outputting information related to the integrated output data, wherein the plurality of groups are classified based on intermediate features output from an intermediate layer of the learning model by inputting a plurality of second learning images each containing an object into a model or base model trained using a first learning image containing the object and correct answer data corresponding to the object, and each of the plurality of object recognizers is a trained model trained using a second learning image from the plurality of second learning images, for which intermediate features corresponding to the group to which each object recognizer belongs are output from the intermediate layer, and correct answer data corresponding to the object included in the second learning image.

[0008] Furthermore, in the object recognition system of the present invention, it is preferable that the learning model is a trained model trained to minimize the error between first learning output data output when a first learning image is input and correct answer data indicating the area of ​​the object within that first learning image, and that each of the multiple object recognizers is a trained model trained to minimize the error between second learning output data output when a second learning image is input, in which intermediate features corresponding to the group to which each object recognizer belongs are output from the intermediate layer, among the multiple second learning images.

[0009] Furthermore, it is preferable that the object recognition system according to the present invention further comprises a classification means for classifying a plurality of groups based on intermediate features, and a learning means for training the plurality of object recognizers so as to minimize an error between second learning output data output when a second learning image, of a plurality of second learning images in which intermediate features corresponding to a group to which each object recognizer belongs are output from the intermediate layer, and ground truth data indicating the area of ​​the object in the second learning image.

[0010] Furthermore, in the object recognition system according to the present invention, it is preferable that the intermediate features are defined by a plurality of feature maps each corresponding to a different component, and the classification means classifies the objects included in the second training image into a plurality of groups based on values ​​representing the features of the plurality of feature maps in the region of the object indicated by the ground truth data of the second training image.

[0011] In addition, in the object recognition system of the present invention, it is preferable that the classification means classifies the objects included in the second training image into multiple groups based on a first vector whose elements are the maximum values ​​of the features of each object included in the second training image in each of the multiple feature maps.

[0012] Furthermore, in the object recognition system according to the present invention, it is preferable that the classification means calculates the positive value of each gradient of the plurality of feature maps in the two intermediate layers, and classifies the objects included in the second training image into a plurality of groups based on a second vector whose elements are the maximum positive values ​​of each object included in the second training image.

[0013] In the object recognition system according to the present invention, the object is preferably a building imaged from above.

[0014] In addition, the object recognition method of the present invention includes obtaining an input image of an object, inputting the input image to a plurality of object recognizers corresponding to each of a plurality of groups, integrating the output data output from each of the plurality of object recognizers, and outputting information related to the integrated output data, wherein the plurality of groups are classified based on intermediate features output from an intermediate layer of the learning model by inputting a plurality of second training images each containing an object into a model or a base model trained using a first training image containing the object and correct answer data corresponding to the object, and each of the plurality of object recognizers is a trained model trained using a second training image from the intermediate layer of which intermediate features corresponding to the group to which each object recognizer belongs are output from the intermediate layer, and correct answer data corresponding to the object included in the second training image.

[0015] Furthermore, a control program according to the present invention is a control program for an object recognition device, and causes the object recognition device to acquire an input image of an object, input the input image to a plurality of object recognizers corresponding to each of a plurality of groups, integrate the output data output from each of the plurality of object recognizers, and output information relating to the integrated output data, wherein the plurality of groups are classified based on intermediate features output from an intermediate layer of a learning model that is a model or base model trained using a first learning image containing the object and correct answer data corresponding to the object, and input a plurality of second learning images each containing an object, and each of the plurality of object recognizers is a trained model trained using a second learning image from the plurality of second learning images, for which intermediate features corresponding to the group to which each object recognizer belongs are output from the intermediate layer, and correct answer data corresponding to the object included in the second learning image. [Effects of the Invention]

[0016] The object recognition system, object recognition method, and control program monitoring system according to the present invention are capable of recognizing objects with higher accuracy. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a configuration diagram of an example of an object recognition system 1. FIG. [Figure 2] 10 is a flowchart illustrating an example of the flow of a learning process. [Figure 3] FIG. 10 is a schematic diagram for explaining a feature amount map. [Figure 4] FIG. 2 is a schematic diagram for explaining a learning recognizer 113. [Figure 5] FIG. 10 is a schematic diagram for explaining a scale. [Figure 6] 10 is a flowchart illustrating an example of the flow of a recognition process. DETAILED DESCRIPTION OF THE INVENTION

[0018] Various embodiments of the present invention will be described below with reference to the drawings. It should be noted that the technical scope of the present invention is not limited to these embodiments, but encompasses the inventions set forth in the claims and their equivalents.

[0019] FIG. 1 is a diagram showing an example of a configuration of an object recognition system 1 according to the present invention.

[0020] The object recognition system 1 includes an object recognition device 100 and a server device 200. The object recognition device 100 and the server device 200 are communicatively connected to each other via a network N. The network N is an intranet, the Internet, or the like.

[0021] The object recognition device 100 is a personal computer, a notebook personal computer, a server, etc. The object recognition device 100 includes an operation device 101, a display device 102, a communication device 103, a storage device 110, a processing circuit 120, etc.

[0022] The operation device 101 has input devices such as a keyboard and a mouse, and an interface circuit that acquires signals from the input devices, accepts operations by a user, and outputs to the processing circuit 120 a signal according to the user's input.

[0023] The display device 102 is an example of an output unit. The display device 102 has a display configured with a liquid crystal display, an organic electroluminescence display, or the like, and an interface circuit that outputs image data to the display, and displays image data on the display in accordance with instructions from the processing circuit 120.

[0024] The communication device 103 is an example of an output unit. The communication device 103 includes a wired or wireless communication interface circuit and connects the object recognition device 100 to a communication network. The communication device 103 performs wired communication in accordance with a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol). The communication device 103 may also perform wireless communication using a wireless communication method conforming to the IEEE (Institute of Electrical and Electronics Engineers) 802.11 standard. The communication device 103 transmits information supplied from the processing circuit 120 to an external device. The communication device 103 also supplies information received from an external device to the processing circuit 120.

[0025] The storage device 110 includes, for example, semiconductor memory such as RAM (Random Access Memory) and ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as an optical disk. The storage device 110 stores computer programs, data, and the like used for processing by the processing circuit 120. The computer programs are installed in the storage device 110 from a server (not shown) via the communication device 103. Note that the computer programs may be installed in the storage device 110 from a computer-readable portable recording medium using a known setup program or the like. The portable recording medium is, for example, a CD-ROM, a DVD-ROM, or the like. The computer programs may be distributed from a server or the like and installed in the storage device 140.

[0026] Furthermore, the storage device 110 stores a learning model 111, a first object recognizer 112-1, a second object recognizer 112-2, ..., an N-th object recognizer 112-N, and a learning recognizer 113. The learning model 111 is used to classify objects into groups, which will be described later. The first object recognizer 112-1, the second object recognizer 112-2, ..., the N-th object recognizer 112-N are examples of multiple object recognizers, and recognize objects from captured images of the objects. Hereinafter, the first object recognizer 112-1, the second object recognizer 112-2, ..., the N-th object recognizer 112-N may be collectively referred to as object recognizer 112. N is a preset integer of 2 or greater, and the object recognizer 112 includes the same number of object recognizers as the number N of groups, which will be described later. The training recognizer 113 is used to generate the object recognizer 112 .

[0027] The processing circuit 120 is, for example, a CPU (Central Processing Unit). The processing circuit 120 may be an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or the like. The processing circuit 120 is connected to the operation device 101, the display device 102, the communication device 103, the storage device 110, and the like, and controls each of these components. The processing circuit 120 reads a program stored in the storage device 110 and operates in accordance with the read program, thereby functioning as a first learning means 121, a classification means 122, a second learning means 123, an acquisition means 124, an integration means 125, and an output control means 126. The processing circuit 120 recognizes an object from a captured image of the object using an object recognizer 112.

[0028] FIG. 2 is a flowchart showing an example of the flow of the learning process executed by the object recognition device 100.

[0029] An example of the operation of the learning process of the object recognition device 100 will be described below with reference to the flowchart shown in Fig. 2. The flow of the operation described below is executed mainly by the processing circuitry 120 in cooperation with each element of the object recognition device 100 based on a program stored in advance in the storage device 110.

[0030] First, the first learning means 121 acquires a first training dataset including a set of a plurality of first training overall images and ground truth data (step S101). The first learning means 121 acquires the first training dataset by receiving it from the server device 200 via the communication device 103. Note that the first training dataset may be pre-stored in the storage device 110, and the first learning means 121 may acquire the first training dataset by reading it from the storage device 110. Each first training overall image is an image of an object. The object may be, for example, a building captured from the sky. In this case, each first training overall image is an aerial photograph or satellite image (or an orthoimage based on the aerial photograph or satellite image) of the ground surface, which is the target area for the building recognition process. The object may be, for example, a road sign, road marking, or vehicle captured on a road, or a suspicious person or intruder captured by a surveillance camera. The ground truth data indicates the shape of the object included in the first training overall image. The correct answer data is, for example, image data that includes the same number of pixels as the number of pixels included in the first learning overall image, and each pixel indicates whether the corresponding pixel in the first learning overall image includes an object. The correct answer data also includes polygon data (vector data in which vertex coordinates are arranged in order) that represents the area of ​​each object and a unique object ID that identifies the object, making it possible to determine the correspondence between each object, the object's area, and the pixels that represent the object. If the object is a building, the correct answer data includes a set of information on the building ID and the building area.

[0031] Next, the first learning means 121 trains the learning model 111 (step S102). The learning model 111 is a model using a neural network and includes an input layer, multiple intermediate layers, and an output layer. The learning model 111 is an object detection model such as R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, SSD (Single Shot MultiBox Detector), or YOLO (You Only Look Once). The first learning means 121 first cuts out multiple window images from each first learning overall image to obtain them as first learning images. The first learning means 121 randomly selects multiple positions from one first learning overall image and cuts out first learning images from each selected position. Note that the first learning means 121 may cut out the first learning images so that the target object is always included.

[0032] Next, the first learning means 121 inputs each first learning image to the input layer of the learning model 111 and compares the first learning output data output from the output layer of the learning model 111 with the correct answer data, thereby training the learning model 111. The first learning means 121 calculates the error between the first learning output data output from the learning model 111 when each first learning image is input and the correct answer data corresponding to each first learning image. Based on the calculated error, the first learning means 121 changes the values ​​of parameters such as weights in the learning model 111 by error backpropagation or the like.

[0033] Each intermediate layer of the learning model 111 outputs intermediate features that indicate the image features of an object included in the first learning image. Each intermediate feature has components corresponding to the width and height directions of the image, as well as components in the channel direction, and is defined by a plurality of feature maps that each correspond to a different channel component.

[0034] FIG. 3 is a schematic diagram for explaining the feature amount map.

[0035] The intermediate feature F1 shown in FIG. 3 is an intermediate feature output from a specific intermediate layer, and the intermediate feature F2 is an intermediate feature output from an intermediate layer located after the specific intermediate layer. The intermediate features F1 and F2 are defined by feature maps M1, M2, ..., ML of multiple channels, respectively. Each feature map includes, for a corresponding component, multiple feature values ​​that indicate the image features of the corresponding pixel in the first training image. In the example shown in FIG. 3, each feature map includes W feature values ​​in the width direction and H feature values ​​in the height direction.

[0036] In this way, the first learning means 121 trains the learning model 111 using the first learning image containing the object and the correct answer data corresponding to the object. In particular, the first learning means 121 trains the learning model 111 so as to minimize the error between the first learning output data output when the first learning image is input and the correct answer data indicating the area of ​​the object in the first learning image.

[0037] In other words, the learning model 111 is a trained model trained using a first learning image containing an object and supervised answer data corresponding to the object. In particular, the learning model 111 is a trained model trained so as to minimize an error between the first learning output data output when the first learning image is input and the supervised answer data indicating the area of ​​the object in the first learning image.

[0038] Next, the classification means 122 acquires a second training dataset including a set of a plurality of second training overall images and ground truth data (step S103). The classification means 122 acquires the second training dataset by receiving it from the server device 200 via the communication device 103. The second training dataset may be pre-stored in the storage device 110, and the classification means 122 may acquire the second training dataset by reading it from the storage device 110. Each second training overall image is an image of an object. If the object is a building imaged from the sky, each second training overall image may be an aerial photograph or satellite image (or an orthoimage based on the aerial photograph or satellite image) of the ground surface, which is the target area for the building recognition process. In this case, the second training dataset includes information on the scale of each second training overall image. The ground truth data indicates the shape of the object included in the second training overall image. The correct answer data is, for example, image data that includes the same number of pixels as the number of pixels included in the second learning overall image, and each pixel indicates whether the corresponding pixel in the second learning overall image includes an object. The correct answer data also includes polygon data that represents the area of ​​each object and a unique object ID that identifies the object, making it possible to determine the correspondence between each object, the object's area, and the pixels that represent the object. If the object is a building, the correct answer data includes a set of information on the building ID and the building area. When the task of the training recognizer 113 (such as object detection) is the same as the task of the training model 111, some or all of the data sets (pairs of training overall images and correct answer data or pairs of training images and correct answer data) in the second training dataset may be the same as the data sets in the first training dataset, or may be entirely different data sets. Also, even when the task of the training recognizer 113 is different from the task of the training model 111, some or all of the training overall images or training images in the second training dataset may be the same as the training overall images or training images in the first training dataset, or may be entirely different data sets. In this case, the correct answer data corresponds to each task.

[0039] Next, the classification means 122 classifies the objects included in each second whole image for learning into a plurality of groups using the learning model 110 learned by the first learning means 121 (step S104).

[0040] The classification means 122 first identifies an object region corresponding to each object in each second learning overall image using each correct answer data. Next, the classification means 122 cuts out and acquires a plurality of window images as second learning images from each second learning overall image. The classification means 122 randomly selects a plurality of positions from one second learning overall image, and cuts out second learning images from each selected position. Note that the classification means 122 may cut out the second learning images so that the objects are always included.

[0041] Next, the classification means 122 inputs each second learning image cut out from each second whole learning image to the learning model 111, and acquires intermediate features output from a predetermined intermediate layer of the learning model 111. Note that the later the intermediate layer is, i.e., the closer it is to the output layer, the more high-level features it outputs, so it is preferable that the intermediate layer from which the intermediate features are acquired is later, i.e., the closer it is to the output layer.

[0042] Next, the classification means 122 calculates image features (hereinafter referred to as classification features) for each object. For each object included in each second learning image, the classification means 122 calculates, as the classification features for each object, values ​​that represent the features of each feature map that constitutes the intermediate features in each object region indicated by the correct answer data of the second entire learning image, i.e., the correct answer data of the second learning image.

[0043] The classification means 122 calculates a first vector whose elements are the maximum values ​​of the feature amounts of each object included in the second learning image in each feature amount map. For each feature amount map constituting the intermediate feature amounts, the classification means 122 identifies a feature amount corresponding to the object region of each object from among the feature amounts included in each feature amount map, and calculates the maximum value of the identified feature amount. Note that if an object is present in multiple second learning images cut out from a single second whole learning image, the classification means 122 calculates the maximum value of the feature amount corresponding to the object from among the feature amounts included in the corresponding feature amount maps output for the multiple second learning images. The classification means 122 calculates a feature vector whose elements are the maximum values ​​calculated for each feature amount map as a first vector, as shown in the following formula (1): First vector = [maximum feature value in the feature map of the first channel, maximum feature value in the feature map of the second channel, …, maximum feature value in the feature map of the Lth channel] (1)

[0044] Furthermore, the classification means 122 calculates a second vector whose elements are the maximum positive gradient values ​​of the feature maps of the channels corresponding to each other in the two hidden layers for each object included in the second learning image. The classification means 122 identifies features corresponding to the object region of each object in each feature map of the intermediate features output from the predetermined first and second hidden layers. The classification means 122 calculates, as a gradient, the difference between corresponding feature values ​​in the feature maps of the channels corresponding to each other in the intermediate features output from the first and second hidden layers, which correspond to the object region of each object. If the calculated gradient is 0 or less (negative value), the classification means 122 calculates a positive value of the gradient by setting the gradient to 0. For each feature map, the classification means 122 calculates the maximum positive gradient value calculated from the object region of each object in each feature map. When each object is present in multiple second training images extracted from a single second whole training image, the classification means 122 calculates the maximum positive gradient value corresponding to the object among the feature amounts included in the feature amount maps of the corresponding channels output for the multiple second training images. The classification means 122 calculates, as a second vector, a feature vector whose elements are the maximum values ​​calculated for each feature amount map, as shown in the following formula (2): Second Vector = [maximum positive gradient value in the feature map of the first channel, maximum feature value in the feature map of the second channel, …, maximum feature value in the feature map of the Lth channel] (2)

[0045] The classification means 122 calculates a first vector or a second vector for each object contained in each second learning image, and calculates the product (inner product or cross product) of the first vector, the second vector, or the first vector and the second vector as a classification feature for each object.

[0046] Next, the classification means 122 classifies the objects included in each second learning image into a predetermined number (N) of groups based on the calculated classification feature values ​​(vectors) using a clustering method such as the k-means method. The classification means 122 may classify the objects using other methods such as DBSCAN, Gaussian Mixture Model, Mean Shift, etc. The classification means 122 associates the object ID of each object with the number (1 to N) of the group into which the object is classified, and stores them in the storage device 110.

[0047] Furthermore, the classification means 122 stores group identification criteria for identifying the group to which each object belongs in association with the group number in the storage device 110. For example, the classification means 122 stores the center of gravity (cluster center) of each group as the group identification criteria in the storage device 110. For example, when the k-means method is used, the center of gravity of each group is the average vector of the classification features (vectors) of the objects belonging to each group.

[0048] In this way, the classification means 122 classifies each object into a plurality of groups based on intermediate features output from the intermediate layer of the learning model 111 by inputting a plurality of second learning images, each of which includes an object, into the learning model 111. That is, the plurality of groups are classified based on intermediate features output from the intermediate layer of the learning model 111 by inputting a plurality of second learning images, each of which includes an object, into the learning model 111.

[0049] By using the learning model 111, the object recognition device 100 can properly classify each object based on image features (classification features) that have appropriate expressions, rather than features arbitrarily determined by humans. Furthermore, by using the learning model 111, the object recognition device 100 can select classification features without specializing them for a particular object. Therefore, with the object recognition device 100, there is no need to change the design even if the object to be recognized changes, and the number of design steps can be reduced.

[0050] In particular, the classification means 122 classifies the objects included in the second learning image into a plurality of groups based on values ​​that represent the feature quantities of a plurality of feature quantity maps in the region of the object indicated by the ground truth data of the second learning image. This allows the object recognition device 100 to appropriately classify the objects regardless of the size of the region and recognize them with high accuracy.

[0051] Furthermore, the classification means 122 classifies the objects included in the second training image into a plurality of groups based on a first vector whose elements are the maximum values ​​of the features of each object included in the second training image in each of the plurality of feature maps. This enables the object recognition device 100 to appropriately classify the objects based on the classification features that strongly represent the object-likeness, and to recognize the objects with high accuracy.

[0052] Furthermore, the classification means 122 calculates the positive value of the gradient of each of the plurality of feature maps in the two intermediate layers of the learning model 111. The classification means 122 classifies the objects included in the second learning image into a plurality of groups based on a second vector whose elements are the maximum positive values ​​of the gradients of the objects included in the second learning image. The more a feature influences the prediction, the larger the gradient of the feature, and the greater the degree to which the feature contributes to the recognition of each object, the larger the value of each element of the second vector. Therefore, by using the gradient of the feature, the object recognition device 100 can appropriately classify objects based on the classification feature that influences the prediction, and recognize them with high accuracy.

[0053] Next, the second learning means 123 trains the training recognizer 113 (step S105).

[0054] FIG. 4 is a schematic diagram for explaining the training recognizer 113. As shown in FIG.

[0055] The training recognizer 113 is a model using a neural network. As shown in FIG. 4, the training recognizer 113 includes a common unit 114 and an individual unit 115. The common unit 114 is commonly trained regardless of the image features of the object, while the individual unit 115 is trained according to the image features of the object. The individual unit 115 includes a first type unit 115-1, a second type unit 115-2, ..., an Mth type unit 115-M. Each type unit is set for each combination of a model type and a scale variation of an image input to the training recognizer 113. For example, if there are two model types and three scale variations and they are combined in a round-robin fashion, the number M of type units 114 is six. The model type indicates the type of neural network in each type unit. The model type includes a model that combines a convolutional layer and a pooling layer in a CNN (Convolutional Neural Network), a model that uses a convolutional layer that performs dilated convolution operations, and the like. The model type may include models other than CNN, such as a Transformer model or a CRF (Conditional Random Field). A reference scale is determined in advance, and scale variations are determined in advance as 1.0, 0.5, 2.0 times the reference scale, etc. The second learning means 123 cuts out a plurality of window images from the second learning overall image as second learning images and inputs them to the learning recognizer 113. At this time, the learning means 123 adjusts the scale of the second learning overall image as necessary before cutting out the second learning images.

[0056] FIG. 5 is a schematic diagram for explaining the scale.

[0057] FIG. 5 shows second learning images R1, R2, and R3, which are window images including buildings C1, C2, and C3. In this example, the scale of the second learning overall image matches the reference scale, and three scale variations are set: 1.0, 0.5, and 2.0 times the reference scale. The second learning image R1 is an image cut out from the second learning overall image acquired by the classification means 122 without changing the scale. The second learning image R2 is an image cut out from the second learning overall image acquired by the classification means 122 after reducing it by thinning out pixels so that the number of pixels in the width and height directions is halved. The second learning image R3 is an image cut out from the second learning overall image acquired by the classification means 122 after enlarging it by linear interpolation or the like so that the number of pixels in the width and height directions is doubled. The second training image R2 is an image with a scale of 0.5 times that of the second training image R1, and the second training image R3 is an image with a scale of 2.0 times that of the second training image R1. As shown in Fig. 5, each second training image has the same number of pixels (Px x Py), and the larger the scale of each second training image, the larger the physical size of the object contained in one pixel. Px and / or Py are, for example, 16, 32, or 64, and are determined in advance when modeling the training detector 113.

[0058] 4, each of the various classification units includes a first feature unit 116-1, a second feature unit 116-2, ..., an Nth feature unit 116-N, etc. Each feature unit corresponds to a group classified based on the image features of the object. That is, the number N of feature units included in each classification unit is the same as the number of groups.

[0059] The common unit 114 includes an input layer and a portion of an intermediate layer of the training recognizer 113, and the individual unit 115 includes a portion of an intermediate layer and an output layer of the training recognizer 113. A second training image cut out from the second entire training image is input to the common unit 114, and information output from the common unit 114 is input to all feature units of all classification units. Each feature unit outputs second training output data related to an object included in the second training image. The second training output data is, for example, image data including the same number of pixels as the number of pixels included in the second training image, with each pixel indicating the probability of an object being present at the position of that pixel.

[0060] The second learning means 123 first enlarges or reduces the second learning overall image as necessary to match the scale of the various separate parts. Note that a plurality of second learning overall images corresponding to the scale of the various separate parts may be prepared in advance, and in step S103, the classification means 122 may acquire a plurality of second learning overall images corresponding to the scale of the various separate parts.

[0061] Next, the second learning means 123 extracts and obtains a plurality of window images as second learning images from each of the scale-adjusted second learning whole images. The second learning means 123 randomly selects a plurality of positions from one second learning whole image, and extracts second learning images from each of the selected positions. Note that the second learning means 123 may extract the second learning images so that an object is always included, or so that the total number of objects per group included in all of the second learning images is equal.

[0062] Next, the second learning means 123 inputs each second learning image to the input layer of the common unit 114 and compares the second learning output data output from each feature unit of the individual unit 115 with the correct answer data, thereby training the learning recognizer 113. When each second learning image is input, the second learning means 123 calculates the error between the second learning output data output from each feature unit (in each individual unit) corresponding to the group of image features of the object included in each second learning image and the correct answer data corresponding to each second learning image. That is, when a second learning image is input in which intermediate features corresponding to the group to which each feature unit belongs are output from the intermediate layer of the learning model 111, the second learning means 123 calculates the error between the second learning output data output from each feature unit and the correct answer data corresponding to each second learning image. Based on the calculated error, the second learning means 123 changes the values ​​of parameters such as weights for each feature unit using backpropagation or the like. In addition, the second learning means 123 accumulates the error to be propagated from the top layer of each feature part to the bottom layer of the common part 114, and based on the accumulated error, changes the values ​​of parameters such as weights in the common part 114 using an error backpropagation method or the like.

[0063] That is, when each second training image is input, the second learning means 123 learns each feature portion so that the error between the second training output data output from the feature portion corresponding to the group of image features of the object contained in each second training image and the correct answer data corresponding to each second training image is minimized. On the other hand, the second learning means 123 does not take into account the second training output data output from the feature portion that does not correspond to the group of image features of the object contained in each second training image. In this way, each feature portion is trained to be suitable for recognizing an object having the corresponding image feature.

[0064] Next, the second learning means 123 acquires an evaluation dataset including a set of multiple evaluation overall images and correct answer data (step S106). The second learning means 123 acquires the evaluation dataset by receiving it from the server device 200 via the communication device 103. The evaluation dataset may be pre-stored in the storage device 110, and the classification means 122 may acquire the evaluation dataset by reading it from the storage device 110. Each evaluation overall image is an image of an object. If the object is a building imaged from the sky, each evaluation overall image may be an aerial photograph or satellite image (or an orthoimage based on the aerial photograph or satellite image) of the ground surface, which is the target area for the building recognition process. The correct answer data indicates the shape of the object included in the evaluation overall image. For example, the correct answer data is image data including the same number of pixels as the number of pixels included in the evaluation overall image, and each pixel indicates whether the corresponding pixel in the evaluation overall image includes the object. Each evaluation overall image includes one or more objects, each having one or more types of image features. The correct answer data also includes polygon data representing the area of ​​each building and a unique building ID that identifies the building. The evaluation dataset also includes information on the scale of each evaluation overall image. The evaluation overall image may be the same as the second learning overall image. Some or all of the data sets in the evaluation dataset may be common to the data sets in the second training dataset, or all of the data sets may be different.

[0065] Next, the second learning means 123 inputs each evaluation whole image to the training recognizer 113 trained in step S105, and acquires evaluation output data output from the training recognizer 113 (step S107).

[0066] The second learning means 123 first enlarges or reduces the evaluation overall image to match the scale of the various separate parts. Note that a plurality of evaluation overall images corresponding to the scale of the various separate parts may be prepared in advance, and the second learning means 123 may acquire a plurality of evaluation overall images corresponding to the scale of the various separate parts in step S106.

[0067] Next, the second learning means 123 cuts out a plurality of window images from each of the scale-adjusted evaluation whole images to obtain them as evaluation images. The second learning means 123 cuts out the evaluation images from the evaluation whole images while shifting the cut-out positions so that the evaluation images do not overlap each other and all areas included in the evaluation whole images are included in one of the evaluation images. Note that the evaluation images may be cut out so that they overlap each other.

[0068] Next, the second learning means 123 inputs each evaluation image to the input layer of the common part 114 and acquires evaluation output data output from each feature part of the individual part 115.

[0069] Next, the second learning means 123 generates binary evaluation data for each of the multiple overall evaluation images and for each feature portion based on the corresponding evaluation output data (step S108). The binary evaluation data is a binary image with the same number of pixels as the overall evaluation image. If the existence probability indicated for each pixel in the evaluation output data is equal to or greater than a predetermined threshold, the second learning means 123 sets a value indicating that an object is included as the pixel value of the corresponding pixel in the binary evaluation data. On the other hand, if the existence probability indicated for each pixel in the evaluation output data is less than the predetermined threshold, the second learning means 123 sets a value indicating that an object is not included as the pixel value of the corresponding pixel in the binary evaluation data. Note that if the evaluation images are cropped so as to overlap with each other, the second learning means 123 sets pixel values ​​for the overlapping positions by comparing the maximum existence probability indicated for corresponding pixels in each evaluation output data with a predetermined threshold. Instead of the maximum value, an average value or a median value may be used, and which value is to be used is determined in advance. In this way, M×N sets of binary evaluation data corresponding to each feature portion are generated by the process of step S108.

[0070] Next, the second learning means 123 identifies the feature part with the highest accuracy for each group classified based on the image features of the object (step S109). The second learning means 123 first uses each correct answer data to identify an object region corresponding to each object in each evaluation whole image.

[0071] Next, the second learning means 123 calculates image features of each identified object. The second learning means 123 inputs each evaluation image cut out from each evaluation entire image into the learning model 111, and acquires intermediate features output from a predetermined intermediate layer of the learning model 111. Next, the second learning means 123 calculates image features of each object included in each evaluation image. Similar to the processing of step S104, the second learning means 123 calculates, for each object included in each evaluation image, a value representing the feature quantity of each feature quantity map in the area of ​​each object indicated by the correct answer data of the evaluation entire image, i.e., the correct answer data of the evaluation image, as the image feature of each object.

[0072] Next, the second learning means 123 identifies the group to which each object belongs based on the group identification criteria stored in step S104. The second learning means 123 identifies the group whose center of gravity stored in step S104 is closest to the image feature of each object as the group to which each object belongs.

[0073] Next, the second learning means 123 calculates a matching rate for each feature part belonging to each group for each group. To this end, the second learning means 123 first calculates, as a matching rate for each feature part, the proportion of pixels in the evaluation binary data based on the evaluation output data output from each feature part belonging to each group for each evaluation overall image that indicate the inclusion of an object belonging to each group in the correct answer data for all evaluation overall images that indicate the inclusion of an object. This calculates M × N matching rates corresponding to each evaluation binary data, i.e., each feature part. The second learning means 123 then identifies the feature part with the highest matching rate in each group as the feature part with the highest accuracy. This identifies N features.

[0074] Next, the second learning means 123 generates the object recognizer 112 by combining the common part 114 with each feature part identified as the feature part with the highest accuracy (step S110), thereby completing the series of steps. To do this, the second learning means 123 first identifies the type part with the highest matching rate for each of the first feature part 116-1, the second feature part 116-2, ..., the Nth feature part 116-N. Then, the second learning means 123 generates each object recognizer by combining the common part 114 with the feature part included in the identified type part for each of the first feature part 116-1, the second feature part 116-2, ..., the Nth feature part 116-N. That is, the second learning means 123 generates the first object recognizer 112-1 by combining the common part 114 with the first feature part 116-1 of the type part with the highest matching rate among the first feature parts 116-1 included in each type of feature part. Similarly, the second learning means 123 generates the Nth object recognizer 112-N by combining the common part 114 with the Nth feature part 116-N of the type part with the highest degree of matching among the Nth feature part 116-N included in the various type parts.

[0075] In the learning detector 113, the common unit 114 may be an extraction unit that extracts image features without using the second learning and outputs them to each feature unit. In this case, the image features extracted by the extraction unit may be, for example, HOG (Histograms of Oriented Gradients) features, SIFT (Scaled Invariance Feature Transform) features, etc. Furthermore, the common part 114 may be omitted from the training recognizer 113. In this case, the second learning means 123 generates an object recognizer so that each of the first feature part 116-1, the second feature part 116-2, ..., the N-th feature part 116-N includes only feature parts included in the identified type part. When the common part 114 is omitted, the second training image (or input image) may be directly input to each feature part.

[0076] In this way, second learning means 123 trains each object recognizer 112 using second learning images in which, among the plurality of second learning images, intermediate features corresponding to the group to which each object recognizer 112 belongs are output from the intermediate layer of learning model 111, and correct answer data corresponding to the object. In particular, second learning means 123 trains each object recognizer 112 so as to minimize the error between second learning output data output when a second learning image in which, among the plurality of second learning images, intermediate features corresponding to the group to which each object recognizer 112 belongs are output from the intermediate layer of learning model 111 is input, and correct answer data indicating the region of the object in that second learning image.

[0077] In other words, each object recognizer 112 is a trained model trained using second training images in which, among the plurality of second training images, intermediate features corresponding to the group to which each object recognizer 112 belongs are output from the intermediate layer of training model 111, and correct answer data corresponding to the object. In particular, each object recognizer 112 is a trained model trained so as to minimize the error between second training output data output when a second training image in which, among the plurality of second training images, intermediate features corresponding to the group to which each object recognizer 112 belongs are output from the intermediate layer of training model 111 is input, and correct answer data indicating the region of the object in that second training image.

[0078] That is, each object recognizer 112 is generated corresponding to each of a plurality of groups classified based on the image features of the objects. The object recognition device 100 generates an object recognizer for each of a plurality of types of image features that can recognize objects having each image feature with high accuracy, and can recognize objects having each image feature with high accuracy.

[0079] FIG. 6 is a flowchart showing an example of the flow of the recognition process executed by the object recognition device 100.

[0080] An example of the operation of the recognition processing of the object recognition device 100 will be described below with reference to the flowchart shown in Fig. 6. The flow of the operation described below is executed mainly by the processing circuit 120 in cooperation with each element of the object recognition device 100 based on a program stored in advance in the storage device 110.

[0081] First, the acquisition means 124 acquires an input overall image in which one or more objects are captured (step S201). The acquisition means 124 acquires the input overall image by receiving it from the server device 200 via the communication device 103. The input overall image is an image in which the object is captured. If the object is a building captured from the sky, the input overall image is an aerial photograph or satellite image (or an orthoimage based on the aerial photograph or satellite image) of the ground surface, which is the target area for processing to recognize the object. The acquired information includes the scale of the input overall image.

[0082] Next, the acquisition means 124 acquires an input image in which one or more objects are captured (step S202). The acquisition means 124 enlarges or reduces the input entire image to match the scale of each object recognizer 112, i.e., the scale of the classification unit that was the basis for each object recognizer 112. Next, the acquisition means 124 cuts out and acquires multiple window images from the scale-adjusted input entire image as input images in which the objects are captured. The acquisition means 124 cuts out the input images from the input entire image while shifting the cut-out positions so that the input images do not overlap each other and so that all areas included in the input entire image are included in one of the input images. Note that the input images may be cut out so that they overlap each other.

[0083] Next, the integration means 125 inputs each input image to each object recognizer 112 (inputs each to all object recognizers 112), and acquires output data output from each object recognizer 112 (step S203).

[0084] Next, the integration means 125 generates overall data by combining the output data from each object recognizer, i.e., the first object recognizer 112-1, the second object recognizer 112-2, ..., the N-th object recognizer 112-N (step S204). The overall data is a multi-valued image whose number of pixels is the same as the number of pixels in the input overall image scaled to each object recognizer 112. The integration means 125 sets the existence probability indicated by the pixel in the corresponding output data as the pixel value of each pixel in the overall data. Note that if the input images are cut out so as to overlap with each other, the integration means 125 sets the maximum value of the existence probabilities indicated by the pixels in each corresponding output data as the pixel value of the pixel in the overall data corresponding to the overlapping position. The average or median may be used instead of the maximum value. However, which value is used is predetermined to be the same as that used when generating evaluation binary data based on the evaluation output data.

[0085] Next, the integration means 125 integrates the generated overall data to generate integrated data (step S205). The integration means 125 first enlarges or reduces each overall data to match the scale of the input overall image. The integration means 125 identifies the maximum pixel value for each corresponding pixel in each overall data, and generates integrated data such that each identified maximum value becomes the pixel value of the corresponding pixel.

[0086] In this way, the integrating means 125 integrates the output data output from each object recognizer 112 to generate integrated data. The integrated data is an example of integrated output data. In other words, "integrating output data" is not limited to simply merging pixel values ​​indicated in multiple output data, but also means combining information included in multiple output data.

[0087] As described above, each object recognizer 112 corresponds to a model type and / or scale capable of recognizing with high accuracy objects having corresponding image features. Therefore, objects having corresponding image features are recognized with high accuracy in the output data output from each object recognizer 112. By integrating the output data output from each object recognizer 112, the integrating means 125 can generate data in which objects having various types of image features are collectively recognized with high accuracy.

[0088] Next, the integration means 125 binarizes the generated integrated data to generate overall binary data (step S206). The overall binary data is a binary image with the same number of pixels as the number of pixels in the integrated data. If the existence probability (maximum value) indicated for each pixel in the integrated data is equal to or greater than a predetermined threshold, the integration means 124 sets a value indicating that the corresponding pixel in the overall binary data contains an object as the pixel value. Furthermore, if the existence probability indicated for each pixel in the integrated data is less than the predetermined threshold, the integration means 124 sets a value indicating that the corresponding pixel in the overall binary data does not contain an object as the pixel value.

[0089] Next, the output control means 126 outputs information about the generated integrated data by transmitting it to an external device via the communication device 103 or by displaying it on the display device 102 (step S207), thereby completing the series of steps. The information about the integrated data is, for example, entire binary data. The information about the integrated data may be the integrated data itself. The information about the integrated data may be information indicating the center of gravity or outer edge position of a pixel group in the integrated data that indicates that the pixel group contains an object.

[0090] As described above, object recognition system 1 inputs an input image to multiple object recognizers corresponding to groups classified based on the image features of the objects, and integrates the output data output from the multiple object recognizers. This enables object recognition system 1 to recognize objects having various image features with high accuracy, making it possible to recognize objects with even higher accuracy.

[0091] Furthermore, the object recognition system 1 can reduce the chances of users overlooking a building when the object is a building imaged from the air. Furthermore, users of the object recognition system 1 can grasp the construction and destruction of buildings, and can obtain basic statistical information on house relocations. Furthermore, users of the object recognition system 1 can more easily grasp the changes in individual buildings over time. Furthermore, users of the object recognition system 1 can easily determine the detailed attributes of a building (e.g., the type of building, such as a detached house, condominium, or factory) from the size and shape of the recognized building area. Furthermore, by automating the process of extracting information about buildings from images, users of the object recognition system 1 can quickly and inexpensively extract information from a wide area of ​​the ground.

[0092] Although preferred embodiments have been described above, the embodiments are not limited to these. For example, in step S107 of FIG. 2, the second learning means 123 may acquire evaluation output data only from feature portions corresponding to image features possessed by objects included in the evaluation images, rather than acquiring evaluation output data from all feature portions. In this case, the second learning means 123 identifies each object included in each evaluation image and identifies the group to which each identified object belongs, similar to the processing in step S109. The second learning means 123 acquires evaluation output data only from feature portions corresponding to the identified group. This allows the object recognition device 100 to reduce the processing load and processing time required to acquire each output data.

[0093] Furthermore, the training method of each object recognizer 112 is not limited to minimizing the error between the second training output data output when a second training image is input and the ground truth data indicating the area of ​​the object having the image feature in the second training image. For example, each object recognizer 112 may be trained to output whether or not an object exists in the second training image when the second training image is input, or the number of objects present, etc.

[0094] Furthermore, the learning method of the learning model 111 is not limited to minimizing the error between the first learning output data output when the first learning image is input and the correct answer data indicating the area of ​​the object in the first learning image. For example, the learning model 111 may be trained to output whether or not an object exists in the first learning image when the first learning image is input, or the number of objects that exist, etc.

[0095] Furthermore, the learning model 111 is not limited to an object detection model, but may be a semantic segmentation model or classification model such as U-NET, MULTISCALE, or HYBRID CNN-CRF. The learning model 111 may also be a so-called foundation model, such as a Segment Anything Model (SAM), that is trained using large amounts of data and applicable to various tasks. The foundation model is trained to be applicable to tasks for many types of arbitrary objects contained in input images. A retrained, i.e., fine-tuned, foundation model may also be used as the learning model 111. In this case, the first learning means 121 inputs each first learning image to an input layer of the foundation model, and trains the foundation model to minimize the error between the first learning output data output from the output layer of the foundation model and the correct answer data, thereby generating the learning model 111.

[0096] In addition, instead of the object recognition device 100, the server device 200 may have a first learning means 121, a classification means 122 and / or a second learning means 123, and execute the learning process of Figure 2 to generate the learning model 111 and / or the object recognizer 112.

[0097] 6, the server device 200 may store the object recognizer 112. In that case, in step S203 of FIG. 6, the integrating means 125 transmits each input image to the server device 200 via the communication device 103. The server device 200 inputs each input image received from the object recognition device 100 to each object recognizer 112 and transmits output data output from each object recognizer 112 to the object recognition device 100. The integrating means 125 acquires the output data by receiving it from the server device 200 via the communication device 103. In that case, the server device 200 may have acquiring means 124 instead of the object recognition device 100, and may execute the processing of steps S201 and / or S202 of FIG. 6 to acquire the entire input image and / or the input image.

[0098] It should be understood by those skilled in the art that various changes, substitutions, and alterations can be made to the present invention without departing from the spirit and scope of the present invention. For example, the above-described embodiments and modifications may be implemented in appropriate combinations within the scope of the present invention. [Explanation of symbols]

[0099] 1 Object Recognition System, 100 Object Recognition Device, 102 Display Device, 103 Communication Device, 112-1 First Object Recognizer, 112-2 Second Object Recognizer, 112-N Nth Object Recognizer, 121 First Learning Means, 122 Classification Means, 123 Second Learning Means, 124 Acquisition Means, 125 Integration Means

Claims

1. an acquisition means for acquiring an input image of an object; an integration means for inputting the input image to a plurality of object recognizers corresponding to each of a plurality of groups, and integrating output data output from each of the plurality of object recognizers; an output means for outputting information relating to the integrated output data, the plurality of groups are classified based on intermediate features output from an intermediate layer of a learning model that is a model or a base model trained using first learning images that include the object and correct answer data corresponding to the object, by inputting a plurality of second learning images that each include the object into the learning model; each of the plurality of object recognizers is a trained model trained using second training images in which intermediate features corresponding to a group to which each object recognizer belongs are output from the intermediate layer among the plurality of second training images, and correct answer data corresponding to an object included in the second training image; An object recognition system characterized by:

2. the learning model is a trained model that has been trained so as to minimize an error between first learning output data that is output when the first learning image is input and correct answer data that indicates a region of an object in the first learning image; 2. The object recognition system according to claim 1, wherein each of the plurality of object recognizers is a trained model that is trained so as to minimize an error between second learning output data that is output when a second learning image, of the plurality of second learning images, in which intermediate features corresponding to a group to which each object recognizer belongs are output from the intermediate layer, and ground truth data that indicates an object region in the second learning image.

3. a classification means for classifying the plurality of groups based on the intermediate feature amount; 3. The object recognition system according to claim 1, further comprising: a learning means for training the plurality of object recognizers so as to minimize an error between second learning output data output when a second learning image, of the plurality of second learning images, in which intermediate features corresponding to a group to which each object recognizer belongs are output from the intermediate layer, and ground truth data indicating an object region in the second learning image.

4. the intermediate features are defined by a plurality of feature maps each corresponding to a different component; 4. The object recognition system according to claim 3, wherein the classification means classifies the objects included in the second training image into the plurality of groups based on values ​​representative of the features of the plurality of feature maps in the region of the object indicated by the correct answer data of the second training image.

5. 5. The object recognition system according to claim 4, wherein the classification means classifies the objects included in the second training image into the plurality of groups based on a first vector whose elements are the maximum values ​​of the features of each object included in the second training image in each of the plurality of feature maps.

6. 6. The object recognition system according to claim 4, wherein the classification means calculates positive values ​​of gradients of the plurality of feature maps in the two intermediate layers, and classifies the objects included in the second training image into the plurality of groups based on a second vector whose elements are the maximum positive values ​​of the objects included in the second training image.

7. The object recognition system according to claim 1 , wherein the object is a building photographed from above.

8. Acquire an input image of the object; inputting the input image to a plurality of object recognizers corresponding to each of a plurality of groups, and integrating output data output from each of the plurality of object recognizers; outputting information about the integrated output data; the plurality of groups are classified based on intermediate features output from an intermediate layer of a learning model that is a model or a base model trained using first learning images that include the object and correct answer data corresponding to the object, by inputting a plurality of second learning images that each include the object into the learning model; each of the plurality of object recognizers is a trained model trained using second training images in which intermediate features corresponding to a group to which each object recognizer belongs are output from the intermediate layer among the plurality of second training images, and correct answer data corresponding to an object included in the second training image; An object recognition method comprising:

9. A control program for an object recognition device, Acquire an input image of the object; inputting the input image to a plurality of object recognizers corresponding to each of a plurality of groups, and integrating output data output from each of the plurality of object recognizers; outputting information about the integrated output data; the plurality of groups are classified based on intermediate features output from an intermediate layer of a learning model that is a model or a base model trained using first learning images that include the object and correct answer data corresponding to the object, by inputting a plurality of second learning images that each include the object into the learning model; each of the plurality of object recognizers is a trained model trained using second training images in which intermediate features corresponding to a group to which each object recognizer belongs are output from the intermediate layer among the plurality of second training images, and correct answer data corresponding to an object included in the second training image; A control program comprising:

Images