Image recognition device, image recognition method

Abstract

To achieve an image recognition device and an image recognition method that enable using a machine learning model obtained in a certain specific domain even in other domain, as to an image recognition-purpose machine learning model.SOLUTION: An image recognition device 100 comprises: an image acquisition unit 110 that acquires an image belonging to any of a plurality of domains; a feature amount extraction unit 120 that extracts a feature amount about a prescribed object from the image; a domain adaptation unit 160 that calculates a domain adaptation feature amount adapting the feature amount to the domain the image belongs to; an auxiliary task unit 150 that calculates an auxiliary feature amount about the object on the basis of the feature amount; and an object recognition unit 130 that recognizes the object from the image on the basis of the feature amount. The image recognition device 100 is configured to conduct learning of the feature amount extraction unit 120, using the domain adaptation unit 160 and the auxiliary task unit 150.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an apparatus and method for realizing the use of learning data between different domains in image recognition for detecting an object from an image by machine learning.

Background Art

[0002] In recent years, image recognition by machine learning has been used to detect a specific object from an image and utilize the detection result for various applications. For example, in on-site work such as the construction industry, various objects existing around a worker are detected using a video camera, and a method for ensuring the safety of the worker by determining whether the working environment is safe based on the detection result has been proposed.

[0003] Generally, in image recognition by machine learning, a machine learning model pre-trained for a specific object is used to identify whether an object existing in an input image is the target object. However, when the machine learning model is obtained for a certain specific domain, if this machine learning model is applied to another domain, the machine learning model does not operate correctly, and thus there is a problem that an appropriate image recognition result cannot be obtained.

[0004] As a method for solving the above problems, a technique called domain adaptation has been devised. Domain adaptation is a technique in which learning is performed so that the difference between an image belonging to a target domain and an image belonging to a source domain becomes small, enabling the machine learning model obtained in the source domain to be applied to a target domain different from the source domain. For example, Patent Document 1 discloses a technique in which domain adaptation is performed between two videos respectively captured by different video cameras, enabling a machine learning model related to human behavior learned from the video of one video camera to be applied to the video of the other video camera.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] U.S. Patent Application Publication No. 2021 / 0064883 [Overview of the project] [Problems that the invention aims to solve]

[0006] The technology described in Patent Document 1 is used in a machine learning model that identifies human behavior from video camera footage, and is not intended for other applications. Therefore, it is difficult to make a machine learning model obtained in one specific domain usable in other domains for image recognition. [Means for solving the problem]

[0007] The image recognition device according to the present invention includes an image acquisition unit that acquires an image belonging to any of a plurality of domains, Having a machine learning model, and using the machine learning model The system comprises: a feature extraction unit that extracts feature quantities relating to a predetermined object from the image; a domain adaptation unit that calculates domain-adapted feature quantities by adapting the feature quantities to the domain to which the image belongs; an auxiliary task unit that calculates auxiliary feature quantities relating to the object based on the feature quantities; and an object recognition unit that recognizes the object from the image based on the feature quantities, wherein the domain adaptation unit Calculation results of the domain-adapted features and the auxiliary task unit Calculation results of the aforementioned auxiliary features The feature extraction unit using The machine learning model in Learn about this. The image recognition method according to the present invention is computer-based, and the computer acquires an image belonging to one of several domains, extracts features relating to a predetermined object from the image using a machine learning model, calculates domain-adapted features by adapting the features to the domain to which the image belongs, calculates auxiliary features relating to the object based on the features, recognizes the object from the image based on the features, and trains the machine learning model using the calculation results of the domain-adapted features and the auxiliary features. [Effects of the Invention]

[0008] According to the present invention, it is possible to realize an image recognition device and an image recognition method in which a machine learning model obtained in a specific domain can be used in other domains as well. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the configuration of an image recognition device relating to one embodiment of the present invention. [Figure 2] This diagram shows the processing details of the feature extraction unit. [Figure 3] This diagram shows the processing details of the object recognition unit. [Figure 4] This diagram shows the processing details of the attention map generation unit and the auxiliary task unit. [Figure 5] This diagram shows the processing details of the domain adaptation section. [Figure 6] This is a flowchart showing the learning process flow in an image recognition device according to one embodiment of the present invention. [Figure 7] This figure shows an example of an information provision screen in an image recognition device according to one embodiment of the present invention. [Figure 8] This figure shows an example of a loss display screen in an image recognition device according to one embodiment of the present invention. [Figure 9] This figure shows an example of a feature display screen in an image recognition device according to one embodiment of the present invention. [Figure 10] This figure shows an example of a network explanation display screen in an image recognition device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. The present invention is not limited to these embodiments, and any application that is consistent with the spirit of the present invention is included within the technical scope of the present invention. Unless otherwise specified, each component may be singular or plural.

[0011] Figure 1 is a block diagram showing the configuration of an image recognition device according to one embodiment of the present invention. As shown in Figure 1, the image recognition device 100 according to this embodiment is connected to an input device 200 and a display device 300. The image recognition device 100 is composed of an image acquisition unit 110, a feature extraction unit 120, an object recognition unit 130, a attention map generation unit 140, an auxiliary task unit 150, a domain adaptation unit 160, a gradient inversion layer 170, and a display control unit 180. In the image recognition device 100, each of these functional blocks is realized, for example, by a computer executing a predetermined program. Note that some or all of these functional blocks may be realized using a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array).

[0012] The image acquisition unit 110 acquires an image to be recognized by the image recognition device 100 and outputs the image data to the feature extraction unit 120. The image to be recognized acquired by the image acquisition unit 110 is, for example, an image obtained by taking a picture of a predetermined location with a camera, and includes predetermined objects such as people, as well as backgrounds other than the objects. The image acquisition unit 110 may acquire a still image, or it may acquire a moving image (video) composed of multiple images (frames) arranged in chronological order. The following explanation will use the case where the image acquisition unit 110 acquires a still image as an example, but when acquiring a moving image, the same processing as for a still image should be performed on each image that makes up the moving image.

[0013] Here, the image acquisition unit 110 acquires an image belonging to one of several domains. A domain corresponds to a classification based on the overall characteristics of the image and differs depending on the camera shooting conditions and the type of background. In other words, the image recognition device 100 of this embodiment receives images belonging to various domains as the image to be recognized and acquires them by the image acquisition unit 110.

[0014] The feature extraction unit 120 extracts features related to a predetermined object from the image of the recognition target acquired by the image acquisition unit 110. The feature extraction unit 120 has a trained machine learning model using a plurality of training images belonging to a predetermined domain (source domain), and by performing predetermined arithmetic processing using this machine learning model, features can be extracted from the image of the recognition target. The features extracted by the feature extraction unit 120 are output to the object recognition unit 130, the attention map generation unit 140, the domain adaptation unit 160, and the display control unit 180, respectively. In the feature extraction unit 120, features may be extracted for various objects by extracting features for each type of object.

[0015] Based on the features extracted by the feature extraction unit 120, the object recognition unit 130 recognizes a predetermined object from the image of the recognition target. The object recognition unit 130 has a trained machine learning model, and using this machine learning model, performs predetermined arithmetic processing on the features extracted by the feature extraction unit 120. Thereby, the region corresponding to the object on the image of the recognition target can be specified, and the region can be recognized as the region where the object is reflected. At this time, a bounding box is set for the specified region of the object. In the feature extraction unit 120, when features are extracted for each type of object as described above, the object recognition unit 130 can recognize various objects from the image of the recognition target by performing the same arithmetic processing on each of the features extracted by the feature extraction unit 120.

[0016] The object recognition unit 130 outputs the recognition result of the object to the display control unit 180 as the object recognition result. Further, for the recognition result of the object, a recognition loss representing the magnitude of the difference from the correct answer is calculated, and the calculation result is output to the display control unit 180 together with the setting result of the above-mentioned bounding box and fed back to the feature extraction unit 120.

[0017] The attention map generation unit 140 identifies areas of interest on the image of the object to be recognized based on the features extracted by the feature extraction unit 120, and generates an attention map representing those areas of interest. An attention map shows where in the image of the object to be recognized the areas of interest that should be considered in the recognition of the object are located, and is generated, for example, by color-coding the image according to the values ​​of the features. The attention map generation unit 140 can also compare the generated attention map with the object recognition result by the object recognition unit 130 to evaluate whether the generation result of the attention map is appropriate. The result of the attention map generation by the attention map generation unit 140 is output to the display control unit 180 and also fed back to the feature extraction unit 120.

[0018] The auxiliary task unit 150 calculates auxiliary features related to a predetermined object based on the features extracted by the feature extraction unit 120. Auxiliary features are features that play an auxiliary role in object recognition and are different from the features extracted by the feature extraction unit 120. The auxiliary task unit 150 has a trained machine learning model, and by using this machine learning model and performing predetermined calculations on the attention map generated by the attention map generation unit 140, it can calculate auxiliary features based on the features.

[0019] The auxiliary task unit 150 calculates an auxiliary task loss, which represents the magnitude of the difference from the correct answer, based on the calculation results of the auxiliary features, and outputs the calculation result to the display control unit 180 and also feeds it back to the feature extraction unit 120.

[0020] The domain adaptation unit 160 calculates domain-adapted features by adapting the features extracted by the feature extraction unit 120 to the domain to which the image to be recognized belongs. As described above, the feature extraction unit 120 extracts features from the image to be recognized using a machine learning model that has been trained using training images belonging to a predetermined source domain. In this case, if the domain to which the image to be recognized belongs is the target domain, and the target domain and the source domain do not match, the features extracted by the feature extraction unit 120 are not necessarily optimal in representing the characteristics of the object. That is, the features extracted by the feature extraction unit 120 from the image belonging to the target domain will include errors corresponding to the difference between the target domain and the source domain. In the image recognition device 100 of this embodiment, domain-adapted features are calculated based on the features so that these errors corresponding to the difference between domains are canceled out. The domain adaptation unit 160 has, for example, a trained machine learning model, and by using this machine learning model and performing predetermined calculations on the features extracted by the feature extraction unit 120, it can calculate domain-adapted features from the features.

[0021] The domain adaptation unit 160 calculates a domain adaptation loss, which represents the magnitude of the difference from the ground truth, based on the calculation results of the domain adaptation features. It outputs the calculation result to the display control unit 180 and also feeds it back to the feature extraction unit 120 via the gradient inversion layer 170. The gradient inversion layer 170 inverts the sign of the domain adaptation loss input from the domain adaptation unit 160 and outputs it to the feature extraction unit 120. This enables backpropagation of errors from the domain adaptation unit 160 to the feature extraction unit 120, realizing adversarial learning of the feature extraction unit 120 based on the domain adaptation loss.

[0022] The display control unit 180 displays a screen on the display device 300, according to the user's instructions, showing the calculation results of each functional block: the feature extraction unit 120, the object recognition unit 130, the attention map generation unit 140, the auxiliary task unit 150, and the domain adaptation unit 160. From the screen displayed on the display device 300, the user can check the recognition results of the object for the image to be recognized, as well as the learning status of the machine learning models of the feature extraction unit 120, the object recognition unit 130, the auxiliary task unit 150, and the domain adaptation unit 160.

[0023] The input device 200 consists of, for example, a keyboard, mouse, or touch panel, and detects user operations and transmits the operation details to the image recognition device 100. The display device 300 consists of, for example, a liquid crystal display, and provides information to the user by displaying various screens according to the control of the display control unit 180. Alternatively, a computer connected to the image recognition device 100 via a communication network may be used as the input device 200 or the display device 300.

[0024] Next, the processing details of the feature extraction unit 120, object recognition unit 130, attention map generation unit 140, auxiliary task unit 150, and domain adaptation unit 160 will be described below with reference to Figures 2 to 5.

[0025] Figure 2 shows the processing details of the feature extraction unit 120.

[0026] The feature extraction unit 120 is configured, for example, using a CNN (Convolutional Neural Network) consisting of multiple layers. It extracts features from the input image by performing convolution operations using trained parameters in each layer. For example, as shown in Figure 2(a), when an image 11 belonging to domain D1 is input to the feature extraction unit 120, the feature quantities 21 of the image 11 are extracted by performing convolution operations on this image 11 in each layer of the feature extraction unit 120. Similarly, as shown in Figure 2(b), when an image 12 belonging to domain D2 is input to the feature extraction unit 120, the feature quantities 22 of the image 12 are extracted by performing convolution operations on this image 12 in each layer of the feature extraction unit 120. The extracted features 21 and 22 each contain information about the characteristics of a specific object (e.g., a dog) in images 11 and 12, respectively.

[0027] Figure 3 shows the processing details of the object recognition unit 130.

[0028] The object recognition unit 130 is configured, for example, using a DNN (Deep Neural Network) consisting of multiple layers. By performing weighting calculations using trained parameters in each layer, it identifies the region of an object from the input features and recognizes the object. For example, as shown in Figure 3, when the feature quantities 22 extracted from the image 12 are input to the object recognition unit 130, weighting calculations are performed on these feature quantities 22 in each layer of the object recognition unit 130, thereby identifying the regions 12a and 12b of the object on the image 12, and the object is recognized in these regions 12a and 12b, respectively. Furthermore, the object recognition unit 130 can also calculate the probability of the object recognition result in the identified regions 12a and 12b, and output the calculation result along with the information of regions 12a and 12b to the display control unit 180.

[0029] Figure 4 shows the processing details of the attention map generation unit 140 and the auxiliary task unit 150.

[0030] The attention map generation unit 140 is configured, for example, using the function gradCAM, and generates an attention map by performing the gradCAM operation on the input features. In this attention map, the focus region corresponding to the foreground part of the image and the background part are displayed separately. The auxiliary task unit 150, similar to the object recognition unit 130, is configured, for example, using a DNN consisting of multiple layers, and calculates auxiliary features from the input attention map by performing weighting calculations using trained parameters in each layer. For example, as shown in Figure 4, when the features 22 extracted from image 12 are input to the attention map generation unit 140, the attention map generation unit 140 performs the gradCAM operation on these features 22, thereby identifying the focus regions 32a and 32b corresponding to the foreground part of image 12, and generating a focus map 32. Furthermore, when the focus map 32 is input to the auxiliary task unit 150, the auxiliary task unit 150 performs weighting calculations on the focus regions 32a and 32b on the focus map 32 in each layer, thereby calculating the auxiliary features 42 of the object in the image 12. In addition, the auxiliary task loss 52 is calculated based on the auxiliary features 42.

[0031] Figure 5 shows the processing details of the domain adaptation unit 160.

[0032] The domain adaptation unit 160, like the object recognition unit 130 and the auxiliary task unit 150, is configured, for example, using a DNN consisting of multiple layers. By performing weighting calculations using trained parameters in each layer, it calculates domain-adapted features from the input features. For example, as shown in Figure 5, when features 22 extracted from image 12 are input to the domain adaptation unit 160, each layer of the domain adaptation unit 160 performs weighting calculations on these features 22 to calculate domain-adapted features 62, in which the features 22 are adapted to the domain D2 to which image 12 belongs as the target domain. Furthermore, a domain adaptation loss 72 is calculated on the domain-adapted features 62.

[0033] Next, the machine learning model training process performed in the image recognition device 100 will be explained below with reference to the flowchart shown in Figure 6.

[0034] Figure 6 is a flowchart showing the learning process in an image recognition device 100 according to one embodiment of the present invention. In the image recognition device 100 of this embodiment, the process shown in the flowchart of Figure 6 is executed at predetermined intervals, thereby training the machine learning model used for image recognition.

[0035] In step S10, the image acquisition unit 110 acquires image data to be recognized. Here, as described above, an image belonging to one of several domains is acquired, and that image data is output to the feature extraction unit 120.

[0036] In step S20, the feature extraction unit 120 extracts features from the image data acquired in step S10. As explained in Figure 2, for example, the feature extraction unit 120 extracts features related to the characteristics of a predetermined object by performing a convolution operation on the image data using trained parameters in each layer of the CNN.

[0037] In step S30, the object recognition unit 130 performs object recognition based on the features extracted in step S20. As explained in Figure 3, for example, each layer of the DNN in the object recognition unit 130 performs weighting calculations on the features using trained parameters, thereby recognizing a predetermined object in the image data acquired in step S10, identifying the region of the object on the image, and calculating the likelihood of the object recognition result in that region.

[0038] In step S40, the object recognition unit 130 calculates the recognition loss for the object recognition result performed in step S30. Here, for example, the recognition result from step S30 is compared with a pre-set correct object recognition result, and based on the comparison result, the recognition loss is calculated such that the value increases as the deviation from the correct answer increases.

[0039] In step S50, the domain adaptation unit 160 performs domain adaptation based on the features extracted in step S20. Here, as explained in Figure 5, for example, the domain adaptation unit 160 calculates domain-adapted features by performing weighting calculations on the features using trained parameters in each layer of the DNN it comprises.

[0040] Furthermore, if the training images used to train the feature extraction unit 120 belong to multiple source domains, it is preferable for the domain adaptation unit 160 to calculate domain-adapted features for each of those multiple source domains. Also, if there are multiple images from which image data is acquired in step S10, the target domain to which each image belongs is not necessarily the same, and each image may belong to a different target domain. Similarly, in such cases, it is preferable for the domain adaptation unit 160 to calculate domain-adapted features for each of those multiple target domains. In other words, it is preferable for the domain adaptation unit 160 to calculate domain-adapted features for each combination of multiple source domains and multiple target domains. This makes it possible to calculate domain-adapted features for various combinations of source domains and target domains.

[0041] In step S60, the domain adaptation unit 160 calculates the domain adaptation loss for the results of the domain adaptation performed in step S50. Here, for example, the features of a pre-defined target domain are compared with the domain adaptation features calculated in step S50, and based on the comparison result, the domain adaptation loss is calculated such that the value increases as the difference between these features increases.

[0042] In step S70, the attention map generation unit 140 generates an attention map based on the features extracted in step S20. Here, as explained in Figure 4, for example, by performing the gradCAM function operation on the features, the region of interest on the image acquired in step S10 is identified, and an attention map representing that region of interest is generated.

[0043] In step S80, the auxiliary task unit 150 performs an auxiliary task based on the features extracted in step S20. Here, as explained in Figure 4, for example, each layer of the DNN in the auxiliary task unit 150 performs a convolution operation on the attention map generated in step S70 using trained parameters to extract auxiliary features related to the auxiliary features of a predetermined object.

[0044] In step S90, the auxiliary task unit 150 calculates the auxiliary task loss for the result of the auxiliary task performed in step S80. Here, for example, the auxiliary features calculated in step S90 are compared with pre-set auxiliary features, and the auxiliary task loss is calculated based on the comparison result such that the larger the difference between them, the higher the value of the auxiliary task loss.

[0045] In step S100, the calculation results of the recognition loss, domain adaptation loss, and auxiliary task loss calculated in steps S40, S60, and S90, respectively, are fed back to the feature extraction unit 120. At this time, the domain adaptation loss is fed back with its sign inverted via the gradient inversion layer 170, as described above.

[0046] In step S110, the feature extraction unit 120 is trained based on the losses fed back in step S100. Here, for example, a total loss is calculated by combining the values ​​of each of the fed-back losses, and the parameters of the machine learning model of the feature extraction unit 120 are adjusted so that the value of the total loss becomes small. This trains the feature extraction unit 120 based on the processing results of the object recognition unit 130, the auxiliary task unit 150, and the domain adaptation unit 160. In addition to training the feature extraction unit 120, the object recognition unit 130, the auxiliary task unit 150, and the domain adaptation unit 160 may also be trained using the losses fed back.

[0047] In step S120, it is determined whether or not to terminate the training of the feature extraction unit 120. If it is determined that the training of the feature extraction unit 120 should continue, the process returns to step S10 and the aforementioned process is repeated. On the other hand, if it is determined that the training of the feature extraction unit 120 should be terminated, the process shown in the flowchart of Figure 6 is terminated. Note that the determination in step S120 can be made based on predetermined termination conditions, such as training termination conditions set in advance by the user.

[0048] In the image recognition device 100 of this embodiment, the process described above is repeatedly executed until a predetermined termination condition is met, thereby training the feature extraction unit 120. As a result, during the training of the feature extraction unit 120, the process of calculating domain-adapted features using the domain adaptation unit 160 in step S50 and the process of calculating auxiliary features using the auxiliary task unit 150 in step S80 are each performed multiple times. This makes it possible to train the feature extraction unit 120 effectively and efficiently.

[0049] Next, specific examples of information provided to the user by the display control unit 180 will be described below with reference to Figures 7 to 10.

[0050] Figure 7 shows an example of an information-providing screen in an image recognition device 100 according to one embodiment of the present invention. The screen 310 shown in Figure 7 is displayed on the display device 300 in response to user operation by the display control unit 180. The screen 310 is a screen for displaying and presenting the learning status and image recognition status of the image recognition device 100 to the user, and has display frames 311, 312, and 313.

[0051] The display frame 311 is for configuring the learning settings of the feature extraction unit 120. By operating the input device 200, the user can select whether or not to use the domain adaptation by the domain adaptation unit 160, the auxiliary tasks by the auxiliary task unit 150, and the attention map generated by the attention map generation unit 140 for learning the feature extraction unit 120. Furthermore, for domain adaptation, the user can specify the source domain and target domain, and for auxiliary tasks, they can select which auxiliary features to use. The user's selections in the display frame 311 are reflected in the learning process shown in the flowchart of Figure 6 in the image recognition device 100.

[0052] The display frame 312 is a section for selecting the type of information to be displayed on the screen 310. By operating the input device 200, the user can select in the display frame 312 to display one of the following: recognition loss, domain adaptation loss, and auxiliary task loss; feature quantities extracted from the image by the feature quantity extraction unit 120; or network description based on the attention map generated by the attention map generation unit 140.

[0053] The display frame 313 is the part for providing information to the user. The image recognition device 100 can provide information to the user regarding the learning status and image recognition status by displaying the information selected by the display frame 312 in the display frame 313 on the display device 300.

[0054] Figure 8 shows an example of a loss display screen in an image recognition device 100 according to one embodiment of the present invention. The screen shown in Figure 8 is displayed in the display frame 313 when the recognition loss, domain adaptation loss, and auxiliary task loss are selected as display targets in the display frame 312 of Figure 7.

[0055] In the screen shown in Figure 8, graph 3131 represents the recognition loss, graph 3132 represents the domain adaptation loss, and graph 3133 represents the auxiliary task loss, showing how each loss changes with respect to the number of training iterations. From graphs 3131 to 3133, it can be seen that as the training of the feature extraction unit 120 progresses, the recognition loss and auxiliary task loss decrease, while conversely, the domain adaptation loss increases.

[0056] Figure 9 shows an example of a feature quantity display screen in an image recognition device 100 according to one embodiment of the present invention. The screen shown in Figure 9 is displayed in the display frame 313 when a feature quantity is selected as the display target in the display frame 312 of Figure 7.

[0057] In the screen shown in Figure 9, graph 3134 shows the distribution of features when the feature extraction unit 120 is trained without using either domain adaptation or auxiliary tasks. In graph 3134, the white circular marker 314 represents a person, the star-shaped marker 315 represents a crane, and the black circular marker 316 represents a warehouse. From graph 3134, it can be seen that when the feature extraction unit 120 is trained without using either domain adaptation or auxiliary tasks, the features extracted by the feature extraction unit 120 are mixed and distributed with features of different objects. As a result, it becomes difficult for the object recognition unit 130 to correctly identify each object.

[0058] In the screen shown in Figure 9, graph 3135 shows the distribution of features when the feature extraction unit 120 is trained using domain adaptation. Markers 314 to 316 in graph 3135 are the same as in graph 3134. From graph 3135, it can be seen that when the feature extraction unit 120 is trained using domain adaptation, the features extracted by the feature extraction unit 120 are distributed separately for each type of object, compared to graph 3134. As a result, the object recognition unit 130 is able to correctly identify each object.

[0059] In the screen shown in Figure 9, graph 3136 shows the distribution of features when the feature extraction unit 120 is trained using both domain adaptation and auxiliary tasks. In graph 3136, markers 314 to 316 are the same as in graphs 3134 and 3135. From graph 3136, it can be seen that when the feature extraction unit 120 is trained using auxiliary tasks in addition to domain adaptation, the features extracted by the feature extraction unit 120 are further separated and distributed according to the type of object, compared to graph 3135. As a result, the object recognition unit 130 can identify each object more accurately.

[0060] Figure 10 shows an example of a network description display screen in an image recognition device 100 according to one embodiment of the present invention. The screen shown in Figure 10 is displayed in the display frame 313 when the network description is selected as the display target in the display frame 312 of Figure 7.

[0061] In the screen shown in Figure 10, explanatory frames 3137a to 3137d represent the recognition results for each person included in the same image. These explanatory frames show the region of each person in the image and the probability value (%) of the recognition result for each person. Explanatory frames 3137a to 3137d can be obtained from the processing results of the object recognition unit 130. Furthermore, the attention regions 3138a to 3138d represent the attention regions on the attention map that correspond to explanatory frames 3137a to 3137d, respectively. In this way, the network explanation makes it possible to clarify the decision-making process of this system.

[0062] According to the embodiment of the present invention described above, the following effects are achieved.

[0063] (1) The image recognition device 100 includes an image acquisition unit 110 that acquires an image belonging to one of several domains, a feature extraction unit 120 that extracts features related to a predetermined object from the image, a domain adaptation unit 160 that calculates domain-adapted features by adapting the features to the domain to which the image belongs, an auxiliary task unit 150 that calculates auxiliary features related to the object based on the features, and an object recognition unit 130 that recognizes the object from the image based on the features, and the feature extraction unit 120 is trained using the domain adaptation unit 160 and the auxiliary task unit 150. In this way, an image recognition device 100 can be realized in which a machine learning model obtained in a specific domain can be used in other domains as well.

[0064] (2) In the training of the feature extraction unit 120, the image recognition device 100 uses a gradient inversion layer 170 to perform adversarial learning based on the domain adaptation loss calculated by the domain adaptation unit 160 for the domain adaptation features. In this way, the training of the feature extraction unit 120 using the domain adaptation unit 160 can be performed effectively.

[0065] (3) The image recognition device 100 includes an attention map generation unit 140 that identifies a region of interest on an image based on its features and generates an attention map representing the region of interest. The auxiliary task unit 150 calculates auxiliary features using the attention map generated by the attention map generation unit 140. In this way, the auxiliary task unit 150 can easily and appropriately calculate auxiliary features for the region of interest corresponding to the foreground portion of the image.

[0066] (4) The domain adaptation unit 160 can calculate domain adaptation features for each combination of multiple source domains and multiple target domains. In this way, it is possible to calculate domain adaptation features for various combinations of source domains and target domains and perform domain adaptation.

[0067] (5) The object recognition unit 130 identifies the region of the object on the image and calculates the probability of the object being recognized in that region. In this way, useful information can be obtained when using the image recognition results from the object recognition unit 130 or when notifying the user.

[0068] (6) In the training of the feature extraction unit 120, the image recognition device 100 performs multiple times each of the following: a first process of calculating domain-adapted features using the domain adaptation unit 160, and a second process of calculating auxiliary features using the auxiliary task unit 150. In this way, the training of the feature extraction unit 120 can be performed efficiently.

[0069] (7) The domain adaptation unit 160 calculates the domain adaptation loss based on the domain adaptation features (step S60). The auxiliary task unit 150 calculates the auxiliary task loss based on the auxiliary features (step S90). The object recognition unit 130 calculates the recognition loss based on the object recognition result (step S40). The image recognition device 100 trains the feature extraction unit 120 based on the domain adaptation loss, auxiliary task loss, and recognition loss fed back from the domain adaptation unit 160, the auxiliary task unit 150, and the object recognition unit 130, respectively (steps S100, S110). In this way, the feature extraction unit 120 can be trained while reliably reflecting the calculation results of the domain adaptation unit 160, the auxiliary task unit 150, and the object recognition unit 130, respectively.

[0070] (8) The image recognition device 100 uses a computer to acquire an image belonging to one of several domains (step S10), extracts features related to a predetermined object from the image using a machine learning model (step S20), calculates domain-adapted features by adapting the features to the domain to which the image belongs (step S50), calculates auxiliary features related to the object based on the features (step S80), and recognizes the object from the image based on the features (step S30). Then, the machine learning model used for feature extraction is trained using the calculation results of the domain-adapted features and auxiliary features (step S110). In this way, an image recognition method can be realized in which a machine learning model obtained in a specific domain can be used in other domains as well.

[0071] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented using any components without departing from the spirit of the invention. The embodiments and modifications described above are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments that can be conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of Symbols]

[0072] 100...Image recognition device, 110...Image acquisition unit, 120...Feature extraction unit, 130...Object recognition unit, 140...Attention map generation unit, 150...Auxiliary task unit, 160...Domain adaptation unit, 170...Gradient inversion layer, 180...Display control unit, 200...Input device, 300...Display device

Claims

1. An image acquisition unit that acquires images belonging to one of several domains, A feature extraction unit having a machine learning model and using the machine learning model to extract features related to a predetermined object from the image, A domain adaptation unit calculates domain-adapted features by adapting the aforementioned features to the domain to which the image belongs. An auxiliary task unit that calculates auxiliary features related to the object based on the aforementioned features, The system includes an object recognition unit that recognizes the object from the image based on the aforementioned feature quantities, An image recognition device that uses the results of the domain adaptation feature calculation by the domain adaptation unit and the results of the auxiliary feature calculation by the auxiliary task unit to train the machine learning model in the feature extraction unit.

2. In the image recognition device according to claim 1, An image recognition device that, in training the machine learning model, performs adversarial learning based on the domain adaptation loss calculated by the domain adaptation unit for the domain adaptation features.

3. In the image recognition device according to claim 1, The system includes an attention map generation unit that identifies a region of interest on the image based on the aforementioned feature quantities and generates an attention map representing the region of interest, The auxiliary task unit is an image recognition device that calculates the auxiliary feature quantities using the attention map.

4. In the image recognition device according to claim 1, The domain adaptation unit is an image recognition device that calculates domain adaptation features for each combination of multiple source domains and multiple target domains.

5. In the image recognition device according to claim 1, The object recognition unit is an image recognition device that identifies the region of the object on the image and calculates the likelihood of the object being recognized in that region.

6. In the image recognition device according to claim 1, An image recognition device that, in training the machine learning model, performs multiple times each of the following: a first process of calculating the domain-adapted features using the domain-adapted unit, and a second process of calculating the auxiliary features using the auxiliary task unit.

7. In the image recognition device according to claim 1, The domain adaptation unit calculates the domain adaptation loss based on the domain adaptation features, The auxiliary task unit calculates the auxiliary task loss based on the auxiliary features, The object recognition unit calculates the recognition loss based on the recognition result of the object, An image recognition device that trains a machine learning model based on the domain adaptation loss, the auxiliary task loss, and the recognition loss, which are fed back from the domain adaptation unit, the auxiliary task unit, and the object recognition unit, respectively.

8. A computer-based image recognition method, The aforementioned computer, Retrieve images belonging to one of multiple domains, Using a machine learning model, feature quantities relating to a predetermined object are extracted from the image. Domain-adapted features are calculated by adapting the aforementioned features to the domain to which the image belongs. Based on the aforementioned features, auxiliary features relating to the object are calculated, Based on the aforementioned feature quantities, the object is recognized from the image. An image recognition method that trains a machine learning model using the results of calculating the domain adaptive features and auxiliary features.

Images