Object recognition system and object recognition method

Abstract

To provide an object recognition system which can reduce costs and time required for collection of teacher data and can also improve accuracy in recognition.SOLUTION: An essential feature amount extraction unit 105 extracts, from target image data based on image data acquired by a predetermined camera 20, as an essential feature amount, a feature amount relating to an element independent of shooting conditions of the camera 20 from among a plurality of feature amounts relating to each of a plurality of elements relating to a subject in the target image data. A database comparison unit 106 compares the essential feature amount with a registered feature amount as an essential feature amount extracted from reference image data based on image data acquired by a camera 20 different from the predetermined camera 20, and specifies the subject based on the comparison result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an object recognition system and an object recognition method. [Background technology]

[0002] To ensure safety in public spaces, images captured by surveillance cameras are monitored to identify targets such as suspicious objects or individuals. Traditionally, surveillance camera images have been visually monitored by security personnel, but this has posed problems, such as a limit to the number of images that can be monitored at one time. In response to this issue, object recognition technology, which uses techniques such as machine learning to automatically recognize desired targets from images, has been attracting attention in recent years.

[0003] Object recognition technology using machine learning enables accurate recognition of objects by generating a trained model that is trained using a large amount of image data of the object to be recognized as training data (training data) for each surveillance camera that has already been installed. However, if a trained model generated using image data acquired by a specific surveillance camera as training data is applied to image data acquired by a different surveillance camera, such as a newly installed surveillance camera, inappropriate recognition results may be obtained.

[0004] To solve the above problem, it is conceivable to collect a large amount of image data acquired by another surveillance camera as training data and retrain the trained model based on that image data. However, this method has the problem that collecting the training data is time-consuming and costly.

[0005] Patent Document 1 discloses an object recognition system that can reduce the time and cost required to collect training data. This object recognition system recognizes vehicles in order to monitor vehicles occupying parking spaces, and performs domain adaptation, which adjusts the distribution of features between image data acquired by surveillance cameras with different viewpoints, in order to properly identify vehicles even when the viewpoint of the surveillance camera relative to the vehicle changes. This eliminates the need to collect large amounts of image data with different viewpoints as training data, thereby reducing the time and cost required to collect training data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2019 / 0065853 Summary of the Invention [Problem to be solved by the invention]

[0007] However, because domain adaptation focuses on correcting the distribution of image features so that the trained model is independent of factors such as the camera's viewpoint, there is no guarantee that it will be able to learn subtle differences in the object to be recognized. As a result, there is a risk that it will not be able to guarantee sufficient recognition accuracy in tasks such as recognizing various bags owned by individuals at baggage claim areas at airports.

[0008] An object of the present disclosure is to provide an object recognition system and an object recognition method that can improve recognition accuracy while reducing the time and cost associated with collecting training data. [Means for solving the problem]

[0009] An object recognition system according to one aspect of the present disclosure is an object recognition system that identifies a subject appearing in target image data based on image data acquired by a specified imaging device, and includes: an extraction unit that extracts, from the target image data, a feature quantity related to an element independent of the imaging conditions of the imaging device, as an essential feature quantity, from among a plurality of feature quantities related to each of a plurality of elements related to the subject appearing in the target image data; and a comparison unit that compares the essential feature quantity with a registered feature quantity, which is the essential feature quantity extracted from reference image data based on image data acquired by an imaging device other than the imaging device, and identifies the subject based on the comparison result. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve recognition accuracy while reducing the time and cost involved in collecting training data. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a functional configuration of an object recognition system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an object recognition system according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating an example of an operating environment in which an object recognition system according to an embodiment of the present disclosure is operated. [Figure 4] 10 is a flowchart illustrating an example of a recognition process. [Figure 5] FIG. 10 is a diagram illustrating an example of processing related to a domain common element. [Figure 6] FIG. 10 is a diagram for explaining an example of a learning method for a domain adaptation network. [Figure 7] FIG. 10 is a diagram for explaining an example of processing related to essential features. [Figure 8] FIG. 1 is a diagram illustrating an example of an element decomposition network. [Figure 9] FIG. 10 is a diagram showing an example of displaying detection results on a display device. [Figure 10]10A and 10B are diagrams illustrating other examples of displaying detection results on a display device. [Figure 11] 10 is a flowchart illustrating a construction process for constructing a database. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] Fig. 1 is a diagram showing the functional configuration of an object recognition system according to an embodiment of the present disclosure. The object recognition system 10 is connected to a camera 20, which is an image capturing device that acquires image data, and a display device 30 that displays various information, via a network 40 so that they can communicate with each other. In the example of Fig. 1, two cameras 20 and one display device 30 are shown, but the numbers of cameras 20 and display devices 30 are not limited to this example. The object recognition system 10, the camera 20, and the display device 30 may be connected by either a wired or wireless method.

[0014] As shown in FIG. 1, the object recognition system 10 includes a user interface 101, a communication unit 102, an image processing unit 103, a domain adaptation unit 104, an essential feature extraction unit 105, a database comparison unit 106, a model learning unit 107, and an estimation unit 108.

[0015] The user interface 101 has a function of receiving various information from the user and a function of outputting various information to the user.

[0016] The communication unit 102 communicates with external devices such as the camera 20 and the display device 30 via the network 40. For example, the communication unit 102 receives image data from the camera 20 and transmits display information to the display device 30.

[0017] The image processing unit 103 performs various image processing on the image data received by the communication unit 102. For example, the image processing unit 103 performs extraction processing to extract partial image data indicating an area in which a specific subject is captured from the image data. The image processing unit 103 may also perform enhancement processing on the image data to enhance a specific subject.

[0018] The domain adaptation unit 104 inputs target image data, which is image data to be used to identify a subject, into a domain adaptation network trained based on image data acquired by each of a plurality of cameras 20 having different shooting conditions (angle of view, background, etc.), and executes domain adaptation processing to extract domain common elements of the target image data. The target image data is data based on image data acquired by one of the cameras 20, and in this embodiment, is partial image data extracted from the image data by the image processing unit 103 in the extraction processing. The domain common elements are feature quantities that indicate characteristics common to the shooting conditions of the cameras 20 of the target image data, and are, for example, vector information.

[0019] The essential feature extraction unit 105 is an extraction unit that extracts, from the target image data, a feature related to an element independent of the shooting conditions of the camera 20 that acquired the target image data, from a plurality of feature related to each of a plurality of elements related to a recognition target that is a subject captured in the target image data, as an essential feature. The essential feature is, for example, vector information.

[0020] The database comparison unit 106 is a comparison unit that compares a database of domain common elements and essential features with the domain common elements and essential features extracted by the domain adaptation unit 104 and the essential feature extraction unit 105, and identifies the recognition target appearing in the target image data based on the comparison result.

[0021] The model learning unit 107 uses image data depicting a predetermined object as training data to generate an object recognition model that has learned the function of estimating whether or not the predetermined subject is depicted in the image data.

[0022] The estimation unit 108 uses the object recognition model generated by the model learning unit 107 to estimate whether or not a predetermined object is captured in predetermined input image data.

[0023] Fig. 2 is a diagram showing an example of the hardware configuration of the object recognition system 10. As shown in Fig. 2, the object recognition system 10 has a processor 151, a memory 152, a communication device 153, an auxiliary storage device 154, an input device 155, and an output device 156, and the hardware components 151 to 156 are connected to each other via a system bus 157 so as to be able to communicate with each other.

[0024] The processor 151 reads a computer program and executes the read computer program to realize the respective functional units 101 to 108 shown in FIG. 1. The memory 152 stores the computer program executed by the processor 151 and various data used by the processor 151. The communication device 153 communicates with external devices such as the camera 20 and the display device 30 shown in FIG. 1. The auxiliary storage device 154 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and permanently stores various data. The databases described above are stored in the auxiliary storage device 154, for example. The input device 155 is, for example, a keyboard, a mouse, or a touch panel, and accepts operations from a user. The output device 156 is, for example, a monitor or a printer, and outputs various data to a user.

[0025] The computer program executed by processor 151 may be recorded on a computer-readable non-transitory recording medium 158. The type of recording medium 158 is not particularly limited, and may be, for example, a flexible disk, CD-ROM, DVD-ROM, hard disk, SSD, optical disk, magneto-optical disk, CD-R, magnetic tape, or a non-volatile memory card. Furthermore, at least a part of the functions realized by the computer program may be realized in hardware, for example, by designing it as an integrated circuit.

[0026] Furthermore, this system may be a physical computer system (one or more physical computers), or a system built on a group of computing resources (multiple computing resources) such as a cloud platform. The computer system or group of computing resources includes one or more interface devices (including, for example, a communication device and an input / output device), one or more storage devices (including, for example, a memory (main memory) and an auxiliary storage device), and one or more processors.

[0027] Fig. 3 is a diagram showing an example of an operating environment in which the object recognition system 10 operates. Fig. 3 shows an example in which the object recognition system 10 operates at a baggage claim area in an airport.

[0028] At baggage claim area 200 at the airport, belt conveyor 201 is provided to transport bag 300, which is baggage transported by airplane, from a back yard to be handed over to its owner. Inspection device 202 is also provided midway along belt conveyor 201 to inspect the contents of bag 300. Inspection device 202 is, for example, an X-ray inspection device, and acquires fluoroscopic image data of the contents of bag 300 without opening bag 300.

[0029] The object recognition system 10 and the display device 30 are installed, for example, in the management department of an airport. The camera 20 is installed in the pick-up location 200 so that the bag 300 is captured in the image data. In the example of FIG. 3, cameras 20A to 20C are installed as the camera 20. The cameras 20A and 20B are installed so that they can capture an image of the bag 300 on the belt conveyor 201. The camera 20C is installed so that it can capture an image of the bag 300 received by the owner. For example, the camera 20C is installed so that it can capture an image of the owner receiving the bag 300 near the belt conveyor 201, or so that it can capture an overhead image of the entire pick-up location 200. It should be noted that additional cameras 20 may be installed as needed.

[0030] The fluoroscopic image data acquired by inspection device 202 is displayed on display device 30 or output device 156 of object recognition system 10. An airport surveillance officer checks the displayed fluoroscopic image data, and if he determines that a dangerous or suspicious object such as a knife is contained in bag 300, he designates bag 300 as a tracking target. One method for designating a tracking target is, for example, to designate bag 300, the contents of which are shown in the fluoroscopic image data from the image data acquired by camera 20A, via input device 155.

[0031] In this case, the object recognition system 10 sets the bag 300 designated as the tracking target as a designated subject. For example, the object recognition system sets information indicating that the bag 300 is a designated subject to an Id that identifies the bag 300. The Id will be described later.

[0032] Furthermore, object recognition system 10 executes a recognition process to identify bag 300 from the image data acquired by cameras 20B and 20C, and outputs the recognition result, which is the result of the recognition process, to output device 156 using user interface 101, or to display device 30 using communication unit 102. At this time, if the identified bag 300 is the same as the specified subject, object recognition system 10 can easily track the specified bag 300 and its owner by superimposing the recognition result on the original image data.

[0033] The object recognition system 10 may also directly cooperate with the inspection device 202 without the intervention of an observer. For example, the estimation unit 108 of the object recognition system 10 estimates whether an object is captured in the fluoroscopic image data using an object recognition model that defines dangerous objects and suspicious objects as predetermined objects. If an object is captured, the estimation unit 108 sets the bag 300 corresponding to the fluoroscopic image data in which the object is captured as the designated subject. In this case, it is possible to reduce the burden on observers or the number of observers, thereby reducing operating costs.

[0034] When object recognition system 10 is applied to a baggage claim area in this way, it is possible to collect a large amount of image data with a conveyor belt as the background, such as image data acquired by cameras 20A and 20B. However, it is difficult to collect a large amount of equivalent images for image data acquired by camera 20C because the shooting conditions, such as the background and angle of view, vary depending on the claim area where camera 20C is installed and its installation location within the claim area. For this reason, when using conventional machine learning techniques, for example, when a new claim area is opened in an airport, even if sufficient images equivalent to those acquired by cameras 20A and 20B are collected, it may be difficult to prepare sufficient images equivalent to those acquired by camera 20C. The following describes the operation and functions of object recognition system 10, which is capable of accurately recognizing a recognition target even in such a situation.

[0035] FIG. 4 is a flowchart illustrating an example of a recognition process in which the object recognition system 10 detects a recognition target.

[0036] In the recognition process, first, the image processing unit 103 of the object recognition system 10 acquires image data acquired by a predetermined camera 20 (in the example of FIG. 3, the newly installed camera 20C) via the communication unit 102, and extracts partial image data indicating an area capturing a predetermined subject (in the example of FIG. 3, the bag 300) from the image data as target image data (step S301). Note that if the original image data from which the target image data is extracted contains multiple predetermined subjects, the image processing unit 103 extracts multiple target image data corresponding to each of the multiple subjects.

[0037] Next, the domain adaptation unit 104 performs domain adaptation processing on the target image data to extract domain common elements from the target image data (step S302). The database comparison unit 106 compares the domain common elements extracted by the domain adaptation unit 104 with a common element database, which is a database related to domain common elements (step S303).

[0038] FIG. 5 is a diagram for explaining the processes of steps S302 and S303 in more detail.

[0039] 5, the common element database 500 stores, for each Id 501 that identifies a bag 300, which is a predetermined subject, registered common elements 502, which are domain common elements extracted from image data showing the bag 300. The image data from which the registered common elements 502 are extracted is reference image data based on image data acquired by at least one of cameras 20A and 20B other than camera 20C, which is the predetermined camera 20. A method for registering the registered common elements 502 in the common element database 500 will be described later with reference to FIG.

[0040] First, in step S302, the domain adaptation unit 104 inputs the target image data 510 to the domain adaptation network 520 that has learned the function of extracting domain common elements, thereby extracting domain common elements 530 from the target image data 510, and inputs the domain common elements 530 to the database comparison unit 106. The domain adaptation network 520 is a trained model that has trained based on image data acquired by the cameras 20A to 20C.

[0041] Fig. 6 is a diagram illustrating an example of a training method for the domain adaptation network 520. When training the domain adaptation network 520 as shown in Fig. 6, new camera image data 601 acquired by the same camera 20C as the camera that acquires the target image data, and old camera image data 602 acquired by cameras 20A and 20B are used as training data. The amount of new camera image data 601 may be small. It is preferable that the amount of old camera image data 602 is large, and may be, for example, all of the usable images.

[0042] In training the domain adaptation network 610, new camera image data 601 and old camera image data 602 are input to the domain adaptation network 610 before training, and the parameters of the domain adaptation network 610 are adjusted using three different loss functions calculated based on the domain common elements 611 output from the domain adaptation network 610, thereby generating a trained domain adaptation network 610.

[0043] In the example of FIG. 6, the three loss functions are a loss function based on cross entropy (Cross Entropy Loss), a loss function based on Hausdorffian distance modified based on d-SNE (T-distributed Stochastic Neighbor Embedding) (VAT (Virtual Adversarial Training) Loss), and a loss function based on discrimination results (Discriminator Loss). The loss function based on cross entropy and the loss function based on Hausdorffian distance modified based on d-SNE are used to calculate the classification results hθ(X s ) The loss function based on the output of the discrimination result is calculated based on the discrimination result obtained by discriminating the domain common elements 611 obtained from the new camera image data 601 and the old camera image data 602 using a discriminator 613.

[0044] Returning to the description of Fig. 5, in step S303, the database comparison unit 106 compares the domain common element 530 with the registered common element 502 registered in the common element database 500 for each Id 501, and calculates the similarity between the domain common element 530 and the registered common element 502 for each Id 501. The database comparison unit 106 generates information indicating the similarity for each Id 502 as the domain comparison result. The similarity is, for example, a classic metric distance such as Euclidean distance.

[0045] Returning to the description of FIG. 4, the database comparison unit 106 determines whether a predetermined accuracy condition regarding the matching rate between the domain common element 530 and the registered common element 502 most similar to the domain common element 530 is satisfied based on the domain comparison result (step S304). In this embodiment, the accuracy condition is that the similarity of the registered common element 502 most similar to the domain common element 530 is higher than a first threshold, and the similarity of the registered common element 502 second most similar to the domain common element is lower than a second threshold. In this case, the similarity may be normalized to a value ranging from 0 to 1. The closer the normalized similarity is to 1, the more similar it is. In this case, the first threshold is, for example, 0.8, and the second threshold is smaller than the first threshold, for example, 0.3.

[0046] The accuracy condition is not limited to the above example, and may be, for example, that the similarity of the registered common element 502 that is most similar to the domain common element 530 is higher than a first threshold value.

[0047] If the accuracy condition is not satisfied, the essential feature extraction unit 105 executes an essential feature extraction process on the target image data to extract essential features from the target image data (step S305). The database comparison unit 106 compares the essential features extracted by the essential feature extraction unit 105 with an essential feature database, which is a database related to essential features (step S306).

[0048] FIG. 7 is a diagram for explaining the processing of steps S305 and S306 in more detail.

[0049] 7, the essential feature database 700 stores, for each Id 701 that identifies a bag 300, which is a predetermined subject, registered features 702, which are essential features extracted from image data showing the bag 300. Id 701 may be the same as Id 501 shown in FIG. 5. The image data from which the registered features 702 are extracted is reference image data based on image data acquired by at least one of cameras 20A and 20B other than camera 20C, which is the predetermined camera 20. A method for registering the registered features 702 in the essential feature database 700 will be described later with reference to FIG. 12.

[0050] First, in step S305, the essential feature extraction unit 105 extracts element decomposition features 730 from the target image data 510 by inputting the target image data 510 to an element decomposition network 720 that has learned the function of extracting element decomposition features.

[0051] The element decomposition network 720 is, for example, an autoencoder neural network. The autoencoder neural network is configured to have a disentanglement characteristic that disentangles feature amounts related to each of a plurality of elements related to a subject captured in image data, and can output element decomposition features that include feature amounts for each element. The element decomposition network 720 (autoencoder neural network) is configured, for example, by a combination of trained beta VAEs (Valuable Auto Encoders).

[0052] FIG. 8 is a diagram showing an example of a component decomposition network configured by a combination of trained beta VAEs. Beta VAEs are known to have component decomposition properties, and can be trained to decompose the features of target image data 510 into color-related features and other features, for example, and output the decomposed features. In this embodiment, as shown in FIG. 8, a component decomposition network 720 configured by a combination of trained beta VAEs outputs, as component decomposition features 730, feature vectors indicating shape-related features related to shape, color-related features related to color, pose-related features related to pose (rotation), and other features related to other elements.

[0053] The essential feature extraction unit 105 discards, from among the element decomposition features 730, the pose-related features that change depending on the shooting conditions of the camera 20C that acquires the target image data 510, and other features as non-essential features 741 that depend on the shooting conditions of the camera 20C, and inputs the color-related features and shape-related features to the database comparison unit 106 as essential features 740 that are features specific to the subject and independent of the shooting conditions of the camera 20C.

[0054] Returning to the description of Fig. 7, in step S306, the database comparison unit 106 compares the essential feature 740 with the registered feature 702 registered in the essential feature database 700 for each Id 701, and calculates the similarity between the essential feature 740 and the registered feature 702 for each Id 701. The database comparison unit 106 generates information indicating the similarity for each Id 702 as the essential comparison result. The similarity is, for example, a classic metric distance such as Euclidean distance.

[0055] Returning to the explanation of Fig. 4, the database comparison unit 106 determines whether the recognition target in the target image is the same as the designated subject (tracking target) based on the domain comparison result generated in step S303 and the essence comparison result generated in step S306. The output unit, which is the user interface 101 or the communication unit 102, outputs the determination result as the recognition result (step S307), and ends the process.

[0056] Specifically, if it is determined in step S304 that the accuracy condition is met, the database comparison unit 106 identifies, based on the domain comparison result, the bag 300 identified by Id 502 corresponding to the registered common element 502 most similar to the domain common element 530, as the recognition target captured in the target image data. On the other hand, if it is determined in step S304 that the accuracy condition is not met, the database comparison unit 106 identifies, based on the essential comparison result, the bag 300 identified by Id 702 corresponding to the registered feature 702 most similar to the essential feature 740, as the recognition target captured in the target image data. Then, the database comparison unit 106 determines whether the recognition target is the same as the set designated subject.

[0057] The determination result may be output by the user interface 101 displaying it on the output device 156 or by the communication unit 102 outputting it to the display device 30. In addition, if the recognition target is the same as the designated subject, the image processing unit 103 may perform enhancement processing on the original image data of the target image data that shows the designated subject, to enhance the designated subject, and output the image data that has undergone the enhancement processing as the determination result.

[0058] 9 and 10 are diagrams showing examples of display of the determination results on the display device 30. FIG.

[0059] The example in Figure 9 is a display example in which image processing unit 103 performs emphasis processing on the image data (image data acquired by camera 20C) that is the source of the target image data, by superimposing a rectangle 31 that surrounds the area where the bag of the designated subject is captured and the area where the person carrying the bag of the designated subject is captured. In this case, the surveillance officer can easily identify the tracking target (designated subject). Note that the emphasis processing may also be performed by surrounding bags other than the designated subject with a rectangle 32 shown with dashed lines, and surrounding the designated subject with a rectangle 31 shown with solid lines.

[0060] 10 is a display example in which multiple image data acquired by multiple cameras, including camera 20C, are simultaneously displayed. In each image data, a rectangle 31 is superimposed, surrounding the area in which the bag of the designated subject is captured.

[0061] The display screen shown in Fig. 9 and the display screen shown in Fig. 10 may be switched in response to an operation by a user such as a security guard. For example, if a touch panel sensor is provided in the display device 30, when the display screen shown in Fig. 9 is tapped, the display screen shown in Fig. 10 may be displayed, and when any of the image data is tapped on the display screen shown in Fig. 10, the tapped image data may be displayed as shown in Fig. 9.

[0062] FIG. 11 is a flowchart for explaining the construction process for constructing a database.

[0063] In the construction process, first, the image processing unit 103 of the object recognition system 10 acquires the old camera image data acquired by the cameras 20A and 20B via the communication unit 102 (step S501).

[0064] The image processing unit 103 checks whether or not a bag, which is a predetermined subject, is captured in the old camera image data (step S502).

[0065] If the bag is not captured, the image processing unit 103 terminates the process. On the other hand, if the bag is captured, the image processing unit 103 extracts partial image data indicating the area in which the bag is captured from the old camera image data as reference image data, and outputs the partial image data to the domain adaptation unit 104 and the essential feature extraction unit 105 (step S503).

[0066] The domain adaptation unit 104 executes domain adaptation processing on the reference image data to extract domain common elements, which are vector information, in the same way as in step S302 in Fig. 4. Furthermore, the essential feature extraction unit 105 executes essential feature extraction processing on the reference image data to extract essential features, which are vector information, in the same way as in step S305 in Fig. 4 (step S504).

[0067] The database comparison unit 106 determines whether the vector information extracted in step S504 has already been registered in the database (step S505). In this embodiment, the vector information used for the determination is an essential feature. In this case, if a registered feature whose similarity (e.g., metric distance) with the essential feature is equal to or greater than a predetermined value is registered in the essential feature database, it may be determined that the vector information has already been registered in the database. Note that the vector information used for the determination may be a domain common element, or may be both a domain common element and an essential feature.

[0068] If the vector information is registered, the database comparison unit 106 ends the process. On the other hand, if the vector information is not registered, the database comparison unit 106 generates a new ID that does not overlap with any ID already registered in the database as an ID for identifying the reference subject appearing in the reference image data, associates the new ID with the domain common element and essential feature extracted in step S504, and registers them in the database (step S506), and ends the process.

[0069] It is also possible to register all of the extracted vector information in the database without performing the process in step S505.

[0070] In this embodiment, the recognition target has been described as a bag, but the recognition target is not limited to a bag. Furthermore, the essential feature can be set appropriately depending on the recognition target, for example. For example, if the recognition target is a human, the essential feature may be a feature related to the color of the clothing. Furthermore, if the recognition target is an animal, the essential feature may be a feature related to the color of the body.

[0071] Furthermore, according to the present embodiment described above, the essential feature extraction unit 105 extracts, from target image data based on image data acquired by camera 20C, feature quantities related to elements independent of the shooting conditions of camera 20C as essential features from among multiple feature quantities related to each of multiple elements related to the subject captured in the target image data. The database comparison unit 106 compares the registered feature quantities, which are essential features extracted from reference image data based on image data acquired by cameras 20A and 20B other than camera 20C, and identifies the subject based on the comparison results. Therefore, because the subject is identified based on feature quantities related to elements independent of the shooting conditions of camera 20C, it is possible to improve recognition accuracy while reducing the time and cost required for collecting training data.

[0072] In this embodiment, the essential feature is at least one of a feature related to the color of the subject and a feature related to the shape of the subject, which makes it possible to extract an appropriate feature as the essential feature.

[0073] In this embodiment, the database comparison unit 106 identifies, as the recognition target, the reference subject corresponding to the registered feature that has the highest similarity to the essential feature in the essential feature database in which registered feature is registered for each reference subject appearing in the reference image data, thereby making it possible to more appropriately identify the recognition target.

[0074] Furthermore, in this embodiment, if the similarity of the registered feature with the highest similarity is higher than a predetermined value, the database comparison unit 106 identifies the reference subject corresponding to the registered feature as the recognition target, thereby enabling more appropriate identification of the recognition target.

[0075] Furthermore, in this embodiment, if the recognition target is the same as the designated subject, the image processing unit 103 performs image processing to emphasize the area in which the designated subject appears in the image data that is the source of the target image data. The user interface 101 or the communication unit 102 outputs the image data that has been subjected to image processing. In this case, it becomes possible for the user to easily identify the designated subject.

[0076] Furthermore, in this embodiment, the database comparison unit 106 registers the essential feature amounts extracted from the reference image data as registered feature amounts in the essential feature amount database. This makes it possible to build and update the database in real time, and to appropriately identify the subject even when recognizing a bag in an airport, for example.

[0077] In this embodiment, the domain adaptation unit 104 extracts domain common elements from the target image data. The database comparison unit 106 identifies the recognition target based on the comparison result of the essential features and the domain comparison result of comparing the domain common elements with registered common elements, which are domain common elements extracted from the reference image data. This makes it possible to more appropriately identify the subject.

[0078] In this embodiment, if a predetermined accuracy condition regarding the matching rate between the registered common element having the highest similarity to the domain common element and the domain common element is satisfied, the database comparison unit 106 identifies the reference object corresponding to the registered common element having the highest similarity as the object, and if the accuracy condition is not satisfied, identifies the object based on the comparison result of the essential feature amount, thereby enabling more appropriate identification of the object.

[0079] In this embodiment, the accuracy condition is that the similarity of the registered common element having the highest similarity to the domain common element is higher than a first threshold, and the similarity of the registered common element having the second highest similarity to the domain common element is lower than a second threshold that is lower than the first threshold, thereby enabling the subject to be more appropriately identified.

[0080] Furthermore, in this embodiment, the database comparison unit 106 registers domain common elements extracted from the reference image data in the common element database. This makes it possible to build and update the common element database in real time, making it possible to appropriately identify the recognition target even when recognizing a bag in an airport, for example.

[0081] The above-described embodiments of the present disclosure are merely illustrative examples of the present disclosure, and are not intended to limit the scope of the present disclosure to these embodiments alone. Those skilled in the art may implement the present disclosure in various other forms without departing from the scope of the present disclosure.

[0082] For example, the number of elements (including the number, numerical value, amount, range, etc.) is not limited to a specific number, and may be greater than or less than a specific number, unless otherwise specified or clearly limited in principle to a specific number. Furthermore, the description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions. Furthermore, the type of existing learning model is arbitrary, such as a deep learning model. [Explanation of symbols]

[0083] 10: Object recognition system 20: Camera 30: Display device 101: User interface 102: Communication unit 103: Image processing unit 104: Domain adaptation unit 105: Essential feature extraction unit 106: Database comparison unit 107: Model learning unit 108: Estimation unit

Claims

1. An object recognition system that identifies a subject appearing in target image data based on image data acquired by a predetermined photographing device, an extracting unit that extracts, from the target image data, a feature quantity related to an element independent of a photographing condition of the photographing device, as an essential feature quantity, from among a plurality of feature quantities related to each of a plurality of elements related to a subject captured in the target image data; a comparison unit that compares the essential feature with a registered feature, which is the essential feature extracted from reference image data based on image data acquired by an image capture device other than the image capture device, and identifies the subject based on the comparison result; a domain adaptation unit that inputs the target image data to a domain adaptation network that has been trained based on image data acquired by the imaging device and the other imaging device, and extracts domain common elements that indicate features common to the imaging conditions of the imaging device and the other imaging device of the target image data, The comparison unit identifies the subject based on the comparison result and a domain comparison result obtained by comparing the domain common element with a registered common element, which is the domain common element extracted from the reference image data.

2. The object recognition system according to claim 1 , wherein the essential feature is at least one of a feature related to a color of the object and a feature related to a shape of the object.

3. 2. The object recognition system according to claim 1, wherein the comparison unit identifies, as the subject, a reference subject corresponding to a registered feature that has the highest similarity to the essential feature in an essential feature database in which the registered feature is registered for each reference subject appearing in the reference image data.

4. The object recognition system according to claim 3 , wherein when the similarity in the registered feature with the highest similarity is higher than a predetermined value, the comparison unit identifies the reference object corresponding to the registered feature as the object.

5. an image processing unit that performs image processing to emphasize an area in which the designated subject appears in image data that is the source of the target image data when the subject is identical to a designated subject that is the reference subject that has been designated in advance; The object recognition system according to claim 3 , further comprising: an output unit that outputs the image data that has been subjected to the image processing.

6. the extraction unit extracts the essential feature from each of the reference image data; The object recognition system according to claim 3 , wherein the comparison unit registers the essential feature extracted from the reference image data as the registered feature in the essential feature database.

7. 2. The object recognition system according to claim 1, wherein the comparison unit identifies, as the subject, the reference subject corresponding to the registered common element having the highest similarity to the domain common element in a common element database in which the registered common elements are registered for each reference subject appearing in the reference image data, if a predetermined accuracy condition regarding a matching rate between the domain common element and the registered common element is satisfied, and identifies the subject based on the comparison result if the accuracy condition is not satisfied.

8. 8. The object recognition system according to claim 7, wherein the accuracy condition is that the similarity of the registered common element having the highest similarity to the domain common element is higher than a first threshold, and the similarity of the registered common element having the second highest similarity to the domain common element is lower than a second threshold that is lower than the first threshold.

9. the domain adaptation unit extracts the domain common elements from each reference image data; The object recognition system according to claim 7 , wherein the comparison unit registers the domain common elements extracted from the reference image data as the registered common elements in the common element database.

10. An object recognition method for an object recognition system that identifies a subject appearing in target image data based on image data acquired by a predetermined photographing device, comprising: extracting, from the target image data, a feature quantity related to an element independent of the photographing conditions of the photographing device from a plurality of feature quantities related to each of a plurality of elements related to a subject appearing in the target image data, as an essential feature quantity; comparing the essential feature with a registered feature, which is the essential feature extracted from reference image data based on image data acquired by an image capture device other than the image capture device; inputting the target image data into a domain adaptation network trained based on image data acquired by the imaging device and the other imaging device, and extracting domain common elements that indicate features common to the imaging conditions of the imaging device and the other imaging device for the target image data; An object recognition method for identifying the subject based on a comparison result between the essential features and registered features and a domain comparison result obtained by comparing the domain common elements with registered common elements that are the domain common elements extracted from the reference image data.

Images