Learning Recognition Device

Abstract

To provide a learning recognition device for realizing learning of a deep neural network (DNN) for object recognition, which does not need a manual annotation to an image for learning, and can reduce power consumption, time, a hardware amount necessary for learning.SOLUTION: A learning recognition device includes an image conversion unit for converting a simulation image into an artificial site image, a prior learned feature extraction unit for outputting a feature point of the converted image, a re-learning feature extraction unit for re-learning difference between both images and outputting a feature point of an artificial site image, an error calculation unit for a feature extraction unit for calculating difference between the feature point outputted by the re-learning feature extraction unit and the feature point outputted by the prior learned feature extraction unit, a coefficient update unit for a feature extraction unit for updating a coefficient of the re-learning feature extraction unit to be used for re-learning on the basis of the difference, and a re-learning identification unit for re-learning an object identification method on the basis of a feature point outputted by a DNN for re-learning the coefficient updated by the coefficient update unit for a feature extraction unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to deep learning. [Background technology]

[0002] With the expected decline in the working population, there is a growing need for the application of AI (artificial intelligence) to recognize objects in the fields of logistics and production. For example, it is necessary to photograph objects moving along a conveyor belt with a camera, and then use AI to recognize the object's type, position, normality, and abnormality from the obtained images.

[0003] Deep learning has been developed in recent years as an AI for image recognition. Deep learning is capable of classifying objects and detecting location information through calculations using a deep neural network (DNN).

[0004] Deep neural networks have an issue with the amount of work required for training. One of the issues is the work required to manually label (annotate) the many images used for training with correct answer information. For this reason, in recent years, methods have been developed in which training images and correct answer information are generated in pairs through simulation, and then used to train DNNs. However, there is an issue that the recognition accuracy deteriorates when a DNN trained with simulated images is used on real images due to differences between simulated images and real images.

[0005] In Patent Document 1, the above problem is solved by converting a real image into an image that imitates a simulation image using a cycle GAN (Generative Adversarial Network, GAN).

[0006] In addition, in Patent Document 2, cycle GAN is used to generate images that imitate real images from simulated images, and the images are used for learning, thereby solving the above problems. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2020-119553 A [Patent Document 2] JP 2020-119554 A Summary of the Invention [Problem to be solved by the invention]

[0008] On the other hand, another issue with deep learning is that traditional backpropagation learning requires a large amount of calculations to learn the entire DNN by working backwards from the back of the DNN, and so the issue is how to reduce the power consumption, time, and hardware required for learning. Thus, the issue with conventional technology is how to reduce the effort required for manual annotation of training images and reduce the power consumption, time, and hardware required for learning.

[0009] One aspect of the present invention aims to realize training of a DNN for object recognition that does not require manual annotation of training images and can reduce the power consumption, time, and amount of hardware required for training. [Means for solving the problem]

[0010] According to one aspect of the present invention, a learning and recognition device includes an image conversion unit that inputs a simulation image and an actual site image into a generative adversarial network and converts the simulation image into an image of an artificial site; a pre-trained feature extraction unit that, at the time of re-learning, inputs the simulation image into a trained deep neural network that has been trained using the simulation image and annotation data for the simulation image and outputs feature points of the simulation image; a re-learning feature extraction unit that inputs the image of the artificial site into a deep neural network for re-learning and re-learns a difference between the simulation image and the image of the artificial site and outputs feature points of the image of the artificial site; an error calculation unit for feature extraction unit that calculates a difference between the feature points output by the re-learning feature extraction unit and the feature points output by the pre-trained feature extraction unit; a coefficient update unit for feature extraction unit that updates coefficients of the re-learning feature extraction unit used for re-learning based on the difference; and a re-learning identification unit that re-learns a method for identifying the item based on feature points output by the deep neural network for re-learning of the coefficients updated by the coefficient update unit for feature extraction unit. Effect of the Invention

[0011] According to one aspect of the present invention, manual annotation of learning images is not required, and the power consumption, time, and amount of hardware required for learning can be reduced. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram showing a first embodiment of the present invention; [Diagram 2] 1 is a diagram showing a time chart of a first embodiment of the present invention; [Diagram 3] 1 is a diagram showing a second embodiment of the present invention; [Figure 4] 11 is a diagram showing a third embodiment of the present invention. [Diagram 5] 13 is a diagram showing a time chart of a third embodiment of the present invention; [Figure 6]11 is a diagram showing a fourth embodiment of the present invention. [Figure 7] 11 is a diagram showing a fifth embodiment of the present invention. [Figure 8] 11 is a diagram showing a sixth embodiment of the present invention. [Figure 9] 13 is a diagram showing a seventh embodiment of the present invention. [Figure 10] 1 is a diagram showing an example of the schematic configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0014] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0015] In the following explanation, various information may be described using expressions such as "database," "table," and "list," but the various information may be expressed in other data structures. To indicate independence of data structure, "XX table," "XX list," and the like may be referred to as "XX information." When describing identification information, when expressions such as "identification information," "identifier," "name," "ID," and "number" are used, these are interchangeable.

[0016] When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts. However, when there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0017] In addition, in the following description, processing performed by executing a program may be described, but the program is executed by a processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) to perform a defined process using storage resources (e.g., a memory) and / or an interface device (e.g., a communication port) as appropriate, so the subject of the processing may be the processor. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing a program may be a calculation unit, and may include a dedicated circuit (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs a specific process.

[0018] A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource for storing a program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0019] The functions and processing in each of the following embodiments can be realized, for example, by a general computer 1600 as shown in FIG. 10 (schematic computer diagram) that includes a CPU 1601, a memory 1602, an external storage device 1603 such as an HDD (Hard Disk Drive), a reading / writing device 1607 that reads and writes information from a portable storage medium 1608 such as a CD (Compact Disk) or a USB memory, an input device 1606 such as a keyboard or a mouse, an output device 1605 such as a display, a communication device 1604 such as a NIC (Network Interface Card) for connecting to a communication network, and an internal communication line (referred to as a system bus) 1609 such as a system bus that connects these together.

[0020] Various data stored in this device or system or used for processing can be realized by the CPU 1601 reading it from the memory 1602 or the external storage device 1603. Also, each functional unit of this device or system (for example, in the first embodiment, the image conversion unit 11, the re-learning feature extraction unit 12, the fixed feature extraction unit 13, the re-learning classification unit 14, the pre-learned feature extraction unit 15, the feature extraction unit error calculation unit 16, the feature extraction unit coefficient update unit 17, the classification unit error calculation unit 18, and the classification unit coefficient update unit 19) can be realized by the CPU 1601 loading a predetermined program stored in the external storage device 1603 into the memory 1602 and executing it.

[0021] The above-mentioned predetermined program may be stored (downloaded) in the external storage device 1603 from the storage medium 1608 via the reading / writing device 1607 or from the network via the communication device 1604, and then loaded onto the memory 1602 and executed by the CPU 1601. Alternatively, the program may be directly loaded onto the memory 1602 from the storage medium 1608 via the reading / writing device 1607 or from the network via the communication device 1604, and executed by the CPU 1601.

[0022] In the following, the present device or system is illustrated as being configured by one computer, but all or part of these functions may be distributed among one or more computers, such as a cloud, and may achieve the same functions by communicating with each other via a network. Also, in FIG. 10, a computer 1600 having a CPU 1601 is illustrated as an example, but the functions and processes in each of the following embodiments may be achieved using a GPU 1610 and an accelerator (dedicated circuit) 1611. EXAMPLES

[0023] The configuration of the first embodiment of the present invention is shown in Fig. 1, and its time chart is shown in Fig. 2. With the configuration in Fig. 1, as shown in Fig. 2, re-learning of an item recognition DNN (Deep Neural Network) and inference using the re-learned item recognition DNN are performed.

[0024] Images for re-learning (simulation images) generated by simulation and associated correct answer information (annotation data) are supplied during re-learning. The simulation images are generated by simulating the shape, pattern, color, etc. of the object to be recognized. The annotation data is data on the item and position information of the object depicted in each simulation image. The position information includes the coordinates, width, and height of the bounding box surrounding the object, and a group of coordinates expressing the outline of the object.

[0025] During re-learning, the simulation images are input to the image conversion unit 11, which converts each simulation image into an image (artificial site image) that mimics the site image including the item to be recognized. The image conversion unit 11 performs the conversion by using a cycle GAN (Generative Adversarial Network) configuration as shown in Patent Document 1 and the like. In order to learn the conversion from the simulation image to an image that mimics the site image, the image conversion unit 11 is also supplied with an actual site image (learning real site image) that includes the item to be recognized. As shown in FIG. 2, the image conversion unit 11 performs the conversion while learning the conversion. As a result, a site image linked to the annotation data can be obtained at the output of the image conversion unit 11, although it is an artificial image (artificial site image).

[0026] During relearning, the artificial site image is input to the DNN for item recognition. The DNN for item recognition comprises, in order, a re-learning feature extraction unit 12, a fixed feature extraction unit 13, and a re-learning identification unit 14. During relearning, the simulation image is also input to the pre-trained feature extraction unit 15. Note that, similarly to the above, the DNN is trained in advance using images and correct answer information generated by simulation, and the part of the front-stage feature extraction unit of the pre-trained DNN model is applied to the pre-trained feature extraction unit 15. The re-learning feature extraction unit 12 also has the same network structure as the pre-trained feature extraction unit 15.

[0027] The pre-trained feature extraction unit 15 outputs feature points corresponding to each of the supplied simulation images. Using these feature points (teacher features), the re-learning feature extraction unit 12 performs re-learning. That is, the feature extraction unit error calculation unit 16 calculates the difference between the feature points output by the re-learned feature extraction unit 12 and the feature points (teacher features) output by the pre-learned feature extraction unit 15. Furthermore, the feature extraction unit coefficient update unit 17 updates the coefficients for the re-learned feature extraction unit 12 based on the difference, and supplies the updated coefficients to the re-learned feature extraction unit 12. The coefficients are weight coefficients and bias coefficients of each neuron present in the re-learned feature extraction unit 12. Note that prior to re-learning, the feature extraction unit coefficient update unit 17 sets the initial coefficients to the same coefficients as those of the pre-learned feature extraction unit 15. As a result, the coefficients of the re-learned feature extraction unit 12 are updated so that the feature points output by the re-learned feature extraction unit 12 are as close as possible to the feature points output by the pre-learned feature extraction unit 15. As a result, the difference between the artificial site image and the simulation image is absorbed in the re-learned feature extraction unit 12. In other words, the output of the re-learned feature extraction unit 12 becomes one that is similar to the feature points of the simulation image.

[0028] As the subsequent fixed feature extraction unit 13, the part of the feature extraction unit subsequent to the feature extraction unit in the previous stage of the pre-trained DNN model is applied. Since this corresponds to a simulation image, the output of the re-training feature extraction unit 12 can be processed as is. Therefore, the coefficients of the fixed feature extraction unit 13 are not re-trained and are left fixed as they are.

[0029] The re-learning feature extraction unit 12 learns the differences between the simulation image and the on-site image, such as the amount of light, contrast, texture, resolution, and noise, while the re-learning classification unit 14 learns a method for identifying the type and position of an object from the feature points output by the fixed feature extraction unit 13. For this purpose, a difference between the output of the re-learning classification unit 14 and the annotation data is calculated by a classification unit error calculation unit 18. A classification unit coefficient update unit 19 updates the coefficients for the re-learning classification unit 14 based on the difference, and supplies the updated coefficients to the re-learning classification unit 14. The coefficients are weight coefficients and bias coefficients of each neuron present in the re-learning classification unit 14. As described above, the coefficients of the re-learning classification unit 14 are updated so that the output of the re-learning classification unit 14 becomes as close as possible to the annotation data.

[0030] As shown in Fig. 2, after re-learning is completed, inference, i.e., recognition, is performed. At this time, the coefficients of the re-learning feature extraction unit 12 and the re-learning identification unit 14 are fixed to the coefficients obtained during re-learning. During inference, the operation of the image conversion unit 11 is not necessary. Therefore, for example, as shown in Fig. 2, the operation of the image conversion unit 11 is turned off to reduce power consumption.

[0031] Furthermore, re-learning can be repeated periodically or irregularly by alternating between re-learning and inference periods, as shown in Figure 2. This makes it possible to follow changes in on-site images caused by changes in lighting, external light, vibration, camera angle of view, camera characteristics, etc., and to adapt at any time to changes or additions of items, or changes in recognition content such as class classification, bounding box detection, and contour detection.

[0032] In this manner, in this embodiment, there are provided an image conversion unit (e.g., image conversion unit 11) that inputs a simulation image and an actual site image into a generative adversarial network and converts the simulation image into an image of an artificial site, a pre-trained feature extraction unit 15 that inputs the simulation image into a trained deep neural network (e.g., a pre-trained DNN model) trained using the simulation image and annotation data for the simulation image during re-learning and outputs feature points of the simulation image, and a deep neural network for re-learning (e.g., a pre-trained DNN model for extracting the feature points of the simulation image). the pre-trained feature extraction unit 16; an error calculation unit for feature extraction unit 16 that calculates the difference between the feature points output by the re-learned feature extraction unit and the feature points output by the pre-trained feature extraction unit; a coefficient update unit for feature extraction unit 17 that updates coefficients of the re-learned feature extraction unit used for re-learning based on the difference; and a re-learning identification unit 14 that re-learns a method for identifying the item based on the feature points output by the deep neural network for re-learning of the coefficients updated by the coefficient update unit for feature extraction unit.

[0033] The system also includes an error calculation unit 18 for the recognition unit that calculates the difference between the output data of the re-learning recognition unit and the annotation data, and a coefficient update unit 19 for the recognition unit that updates the coefficients used by the re-learning recognition unit based on the difference, and the re-learning recognition unit re-learns the method of recognizing the item based on feature points output by the re-learning deep neural network of the coefficients updated by the coefficient update unit for the recognition unit.

[0034] Also, as shown in FIG. 2, the image conversion unit stops conversion of the simulation image into the image of the artificial scene by the generative adversarial network during a period in which the re-learning is not being performed.

[0035] As described above, according to this embodiment, it is possible to realize an AI that can recognize on-site images with high accuracy without the need for manual annotation of learning images and can flexibly adapt to changes in the environment and tasks. In addition, since the re-learning of the feature extraction unit is performed only in the previous stage (for example, the re-learning feature extraction unit 12), it is possible to reduce the power consumption, time, and hardware amount required for re-learning. EXAMPLES

[0036] The configuration of the second embodiment of the present invention is shown in Fig. 3. In this embodiment, the re-learning feature extraction unit in the first embodiment is trained as many times as necessary, starting from the previous layer. As in the first embodiment, images for re-learning (simulation images) generated by simulation and associated correct answer information (annotation data) are supplied during re-learning.

[0037] During re-learning, the simulation images are input to the image conversion unit 31, which converts each simulation image into an image (artificial site image) that imitates the site image. The image conversion unit 31 performs the conversion by using a cycle GAN configuration. To learn the conversion from the simulation image to an image that imitates the site image, the image conversion unit 11 is also supplied with an actual site image (learning real site image) (not shown). The image conversion unit 31 executes the conversion while learning the conversion. As a result, a site image linked to the annotation data can be obtained at the output of the image conversion unit 31, although it is an artificial image (artificial site image).

[0038] During re-learning, the artificial site image is input to the DNN for item recognition. The DNN for item recognition is configured such that the first layer 32 of the re-learning feature extraction unit to the final layer 33 of the re-learning feature extraction unit and the re-learning identification unit 34 are connected in series. During re-learning, the simulation image is also input to the pre-trained feature extraction unit. The pre-trained feature extraction unit is configured such that the first layer 35 of the pre-trained feature extraction unit to the final layer 38 of the pre-trained feature extraction unit are connected in series.

[0039] Note that a DNN is trained in advance using images and correct answer information generated by simulation, and the first to final layers of the feature extraction unit in the pre-trained DNN model are applied to the pre-trained feature extraction unit first layer 35 to the pre-trained feature extraction unit final layer 38. In addition, the re-trained feature extraction unit first layer 32 to the re-trained feature extraction unit final layer 33 also have the same network structure as the pre-trained feature extraction unit first layer 35 to the pre-trained feature extraction unit final layer 38.

[0040] Each layer of the pre-trained feature extraction unit outputs feature points in each layer corresponding to each of the supplied simulation images. Using the feature point groups (teacher features) in each layer, re-learning is performed for each layer of the re-training feature extraction unit. At that time, re-learning is performed for the required number of layers, starting from the first layer.

[0041] When the first layer 32 of the re-learning feature extraction unit is re-learned, the difference between the feature points output by the first layer 32 of the re-learning feature extraction unit and the feature points (teacher features) output by the first layer 35 of the pre-learning feature extraction unit is calculated by the error calculation unit 36 ​​for the feature extraction unit. Furthermore, the coefficient update unit 37 for the feature extraction unit updates the coefficients for the first layer 32 of the re-learning feature extraction unit based on the difference, and supplies the updated coefficients to the first layer 32 of the re-learning feature extraction unit. The coefficients are weight coefficients and bias coefficients of each neuron present in the first layer 32 of the re-learning feature extraction unit. Prior to re-learning, the coefficient update unit 37 for the feature extraction unit sets the initial coefficients to the same coefficients as those of the first layer 35 of the pre-learning feature extraction unit. As described above, the coefficients of the first layer 32 of the re-learning feature extraction unit are updated so that the feature points output by the first layer 32 of the re-learning feature extraction unit are as close as possible to the feature points output by the first layer 35 of the pre-learning feature extraction unit. As a result, the difference between the artificial scene image and the simulation image is somewhat absorbed in the first layer of the re-learning feature extraction unit 32. That is, the output of the first layer of the re-learning feature extraction unit 32 becomes feature points somewhat closer to the feature points of the simulation image (teacher features (first layer)).

[0042] During (or after) relearning of the first layer 32 of the relearning feature extraction unit, the relearning identification unit 34 is relearned. At that time, in each layer from the second layer onwards of the relearning feature extraction unit, the same coefficients as those in the corresponding layers of the pre-trained feature extraction unit are set by the feature extraction unit coefficient update unit of each layer, and the coefficients are fixed as they are.

[0043] The re-learning feature extraction unit learns the differences between the simulation image and the on-site image, such as the amount of light, contrast, texture, resolution, and noise, while the re-learning classification unit 34 learns a method for identifying the type and position of an object from the feature points output by the re-learning feature extraction unit. To this end, a difference between the output of the re-learning classification unit 34 and the annotation data is calculated by a classification unit error calculation unit 311. A classification unit coefficient update unit 312 updates the coefficients for the re-learning classification unit 34 based on the difference, and supplies the updated coefficients to the re-learning classification unit 34. The coefficients are weight coefficients and bias coefficients of each neuron present in the re-learning classification unit 34. As described above, the coefficients of the re-learning classification unit 34 are updated so that the output of the re-learning classification unit 34 is as close as possible to the annotation data.

[0044] After the re-learning in the re-learning classification unit 34 has converged, the accuracy determination unit 313 calculates the current recognition accuracy. The accuracy determination unit 313 is supplied with the difference between the output of the re-learning classification unit 34 and the annotation data, calculated by the classification unit error calculation unit 311, and calculates the current recognition accuracy based on this.

[0045] If the calculated current recognition accuracy is equal to or greater than the target value, re-learning of the DNN for object recognition (i.e., the re-learning feature extraction unit and the re-learning classification unit) is completed. On the other hand, if the current recognition accuracy is less than the target value, the process proceeds to re-learning the second layer of the re-learning feature extraction unit. This is because re-learning the first layer alone will determine that the feature points output by the re-learning feature extraction unit still deviate from the teacher features (teacher features (final layer) in Figure 3).

[0046] When re-learning the second layer of the re-learning feature extraction unit, the coefficients of the first layer 32 of the re-learning feature extraction unit are fixed to the coefficients obtained during re-learning. Re-learning of the second layer of the re-learning feature extraction unit is performed in the same configuration and manner as re-learning of the first layer. That is, the difference between the feature points output by the second layer of the re-learning feature extraction unit and the feature points (teacher features) output by the second layer of the pre-learned feature extraction unit is calculated by the error calculation unit for the feature extraction unit of the second layer. Furthermore, the coefficient update unit for the feature extraction unit of the second layer updates the coefficients for the second layer of the re-learning feature extraction unit based on the difference, and supplies the updated coefficients to the second layer of the re-learning feature extraction unit. The coefficients are weight coefficients and bias coefficients of each neuron present in the second layer of the re-learning feature extraction unit. Prior to re-learning, the coefficient update unit for the feature extraction unit of the second layer sets the initial coefficients to the same coefficients as those of the second layer of the pre-learned feature extraction unit. As a result, the coefficients of the second layer of the re-learned feature extraction unit are updated so that the feature points output by the second layer of the re-learned feature extraction unit are as close as possible to the feature points output by the second layer of the pre-trained feature extraction unit. This allows the difference between the artificial site image and the simulation image to be further absorbed in the second layer of the re-learned feature extraction unit. In other words, the output of the second layer of the re-learned feature extraction unit becomes feature points that are even closer to the feature points of the simulation image (teacher features (second layer)).

[0047] During (or after) relearning the second layer of the relearning feature extraction unit, relearning of the relearning classification unit 34 is performed again. At that time, the coefficients of the third layer and subsequent layers of the relearning feature extraction unit are kept fixed. Relearning of the relearning classification unit 34 is performed in the same manner as described above.

[0048] After the re-learning in the re-learning identification unit 34 has converged, the accuracy determination unit 313 calculates the current recognition accuracy again in the same manner as described above. If the calculated current recognition accuracy is equal to or greater than the target value, the re-learning of the DNN for item recognition is completed. On the other hand, if the current recognition accuracy is less than the target value, it is determined that the feature points output by the re-learning feature extraction unit are still deviated from the teacher features (teacher features (final layer)) by only re-learning the first and second layers. Therefore, the re-learning of the third and subsequent layers of the re-learning feature extraction unit is continued in the same manner until the recognition accuracy calculated by the accuracy determination unit 313 is equal to or greater than the target value. When the recognition accuracy is equal to or greater than the target value, the re-learning of the DNN for item recognition is terminated and the re-learning is completed.

[0049] In the above description, the updating and fixing of coefficients in each layer of the relearning feature extraction unit, that is, turning relearning ON or OFF, is performed by the accuracy determination unit 313 issuing an instruction to the feature extraction unit coefficient update unit of each layer.

[0050] As in the first embodiment, after re-learning is completed, inference, i.e., recognition, is performed. At this time, the coefficients of the re-learning feature extraction unit and the re-learning identification unit are fixed to the coefficients obtained during re-learning. During inference, the operation of the image conversion unit 31 is not necessary, so its operation is turned off to reduce power consumption. Also, as shown in FIG. 2, re-learning and inference may be performed alternately.

[0051] In this embodiment, instead of providing each layer with its own coefficient update unit for the feature extraction unit and / or error calculation unit for the feature extraction unit, each layer may share one or a small number of coefficient update units for the feature extraction unit and / or error calculation unit for the feature extraction unit, since each layer updates its coefficients at different periods.

[0052] In this embodiment, instead of comprehensively re-learning the first, second, and third layers of the feature extraction unit and the previous layers, only effective layers from the previous layers, such as the first, third, and sixth layers, may be re-learned. The effective layers may be selected, for example, based on the user's experience or based on past learning results. In addition, a block such as the first layer 32 of the re-learning feature extraction unit in FIG. 3 may be a block including not only one layer but also multiple layers. For example, the first layer of the re-learning feature extraction unit and the second layer of the re-learning feature extraction unit may be one first layer of the re-learning feature extraction unit. In addition, as in the first embodiment, the rear stage of the re-learning feature extraction unit may be a fixed feature extraction unit.

[0053] Thus, in this embodiment, the trained deep neural network is composed of multiple learning layers, the deep neural network for relearning is composed of multiple re-learning layers, the pre-trained feature extraction unit performs the learning in sequence starting from the earlier layers of the multiple learning layers, and the relearning feature extraction unit performs the relearning in sequence starting from the earlier layers of the multiple relearning layers.

[0054] The re-learning feature extraction unit also includes an accuracy determination unit 313 that determines the accuracy of the deep neural network for re-learning, and terminates re-learning based on the accuracy determination result by the accuracy determination unit.

[0055] The system also includes an error calculation unit for classification unit 311 that calculates the difference between the output data of the re-learning classification unit and the annotation data, and a coefficient update unit for classification unit 312 that updates the coefficients used by the re-learning classification unit based on the difference, and the re-learning classification unit re-learns the classification method for the item based on feature points output by the re-learning deep neural network of the coefficients updated by the coefficient update unit for classification unit.

[0056] As described above, according to this embodiment, it is possible to realize an AI that can recognize on-site images with high accuracy without the need for manual annotation of learning images, and can flexibly adapt to changes in the environment and tasks. In addition, since the feature extraction unit is retrained only for the necessary parts, it is possible to reduce the power consumption, time, and hardware amount required for retraining. Furthermore, the necessary parts can be automatically identified by the accuracy determination unit. EXAMPLES

[0057] The configuration of the third embodiment of the present invention is shown in FIG. 4, and its time chart is shown in FIG. 5. In this embodiment, the learning of the feature extraction unit in the second embodiment is performed in a time-division manner, thereby halving the hardware required to implement the feature extraction unit. As in the second embodiment, an image for re-learning (simulation image) generated by simulation and the associated correct answer information (annotation data) are supplied during re-learning. In the following, an example is shown in which the learning of the feature extraction unit in the second embodiment is performed in a time-division manner, but the same can be considered for the case in which the learning of the feature extraction unit in the first embodiment is performed in a time-division manner.

[0058] During re-learning, the simulation images are input to the image conversion unit 41, which converts each simulation image into an artificial site image that imitates the site image. The image conversion unit 41 performs the conversion using a cycle GAN configuration. To learn the conversion from the simulation image to an image that imitates the site image, an actual site image (real site image) is also supplied to the image conversion unit 41. The image conversion unit 41 executes the conversion while learning the conversion. As a result, a site image linked to the annotation data can be obtained at the output of the image conversion unit 41, although it is an artificial image.

[0059] During re-learning, the simulation image and the artificial site image are input to the DNN for item recognition. The DNN for item recognition is configured such that the feature extraction unit first layer 43 to the feature extraction unit final layer 44 and the re-learning identification unit 45 are connected in series.

[0060] In this embodiment, the pre-trained feature extraction unit and the re-learned feature extraction unit are implemented on one feature extraction unit. This is because the two have the same network structure, although their coefficients are different from each other. Therefore, during re-learning, the coefficients set in the feature extraction unit are alternately switched between coefficients for the pre-trained feature extraction unit (pre-learned coefficients) and coefficients for the re-learned feature extraction unit (re-learned coefficients), thereby alternately realizing the pre-trained feature extraction unit and the re-learned feature extraction unit. At that time, the switching unit 42 alternately switches the image input to the feature extraction unit first layer 43 between the simulation image and the artificial site image.

[0061] The DNN is trained in advance using images and correct answer information generated by simulation, and the first to final layers of the feature extraction unit of the pre-trained DNN model are applied to the first to final layers of the pre-trained feature extraction unit. For this reason, the feature extraction unit coefficient update unit of each layer stores the coefficients of the corresponding layer obtained in the pre-training as pre-trained coefficients, or can call them from a memory (not shown). The feature extraction unit coefficient update unit of each layer also stores the coefficients (relearning coefficients) of the corresponding layer of the re-learning feature extraction unit, and the initial values ​​before re-learning are set equal to the pre-learned coefficients.

[0062] As in the second embodiment, the re-learning is performed for the necessary number of layers, starting from the first layer of the re-learning feature extraction unit, using the teacher features of each layer. As shown in FIG. 5, when re-learning the first layer of the re-learning feature extraction unit, a simulation image is first input to the feature extraction unit first layer 43 by the switching unit 42. At this time, as shown in the figure, the feature extraction unit coefficient update unit 48 for the first layer sets the pre-learned coefficients for the first layer stored as described above to the feature extraction unit first layer 43. That is, the feature extraction unit first layer 43 operates as the first layer of the pre-learned feature extraction unit. That is, the feature points output by the feature extraction unit first layer 43 correspond to the feature points (teacher features (first layer)) output by the first layer of the pre-learned feature extraction unit. Therefore, these are stored in the storage unit 46.

[0063] Next, as shown in Fig. 5, an artificial site image corresponding to the simulation image (i.e., the output of the image conversion unit 41) is input to the feature extraction unit first layer 43. At this time, as shown in the figure, the feature extraction unit coefficient update unit 48 of the first layer sets the re-learning coefficient for the first layer stored as described above to the feature extraction unit first layer 43. That is, the feature extraction unit first layer 43 operates as the first layer of the re-learning feature extraction unit. In other words, the feature points output by the feature extraction unit first layer 43 correspond to the feature points output by the first layer of the re-learning feature extraction unit.

[0064] To update the coefficients in the same manner as in the second embodiment, a feature extraction error calculation unit 47 calculates the difference between the feature points output by the feature extraction unit first layer 43 and the feature points held in the holding unit 46. Furthermore, a feature extraction unit coefficient update unit 48 updates the re-learning coefficients for the first layer based on the difference, and stores the updated coefficients.

[0065] By alternately repeating the above operations, the re-learning coefficients for the first layer are updated so that the feature points output by the first layer of the re-learning feature extraction unit are as close as possible to the feature points output by the first layer of the pre-learned feature extraction unit. As a result, the difference between the artificial site image and the simulation image is somewhat absorbed in the feature extraction unit first layer 43. That is, the output of the feature extraction unit first layer 43 becomes feature points somewhat closer to the feature points of the simulation image (teacher features (first layer)).

[0066] During (or after) relearning of the first layer of the relearning feature extraction unit, relearning of the relearning identification unit 45 is performed. At that time, of the outputs of the feature extraction unit first layer 43, the output as the first layer of the relearning feature extraction unit is transmitted to the second layer of the feature extraction unit. Also, each layer after the second layer of the feature extraction unit is operated as each layer of the pre-trained feature extraction unit. That is, the feature extraction unit coefficient update unit of each layer sets pre-trained coefficients in the feature extraction unit of each layer.

[0067] The re-learning feature extraction unit learns the differences between the simulation image and the on-site image, such as the amount of light, contrast, texture, resolution, and noise, while the re-learning classification unit 45 learns a method for identifying the type and position of an object from the feature points output by the feature extraction unit final layer 44. To this end, the difference between the output of the re-learning classification unit 45 and the annotation data is calculated by the classification unit error calculation unit 412. The classification unit coefficient update unit 413 updates the coefficients for the re-learning classification unit 45 based on the difference, and supplies the updated coefficients to the re-learning classification unit 45. The coefficients are weight coefficients and bias coefficients of each neuron present in the re-learning classification unit 45. As described above, the coefficients of the re-learning classification unit 45 are updated so that the output of the re-learning classification unit 45 is as close as possible to the annotation data.

[0068] After the re-learning in the re-learning classification unit 45 has converged, the accuracy determination unit 414 calculates the current recognition accuracy. The accuracy determination unit 414 is supplied with the difference between the output of the re-learning classification unit 45 and the annotation data, calculated by the classification unit error calculation unit 412, and calculates the current recognition accuracy based on this.

[0069] If the calculated current recognition accuracy is equal to or greater than the target value, the re-learning is completed. On the other hand, if the current recognition accuracy is less than the target value, the process proceeds to re-learning the second layer of the re-learning feature extraction unit. This is because re-learning the first layer alone is judged to still deviate from the teacher features (teacher features (final layer) in FIG. 3).

[0070] Re-learning of the second layer of the feature extraction unit is performed in the same configuration and manner as re-learning of the first layer. As shown in FIG. 5, first, a simulation image is input to the first layer 43 of the feature extraction unit by the switching unit 42. At this time, as shown in the figure, the coefficient update unit 48 for the feature extraction unit of the first layer sets the pre-learned coefficient for the first layer to the first layer 43 of the feature extraction unit. That is, the first layer 43 of the feature extraction unit operates as the first layer of the pre-learned feature extraction unit. Also, the coefficient update unit for the feature extraction unit of the second layer sets the pre-learned coefficient for the second layer to the second layer of the feature extraction unit. That is, the second layer of the feature extraction unit operates as the second layer of the pre-learned feature extraction unit. Therefore, the feature points output by the second layer of the feature extraction unit correspond to the feature points (teacher features (second layer)) output by the second layer of the pre-learned feature extraction unit. Therefore, these are stored in the storage unit of the second layer.

[0071] Next, as shown in FIG. 5, an artificial site image corresponding to the simulation image (i.e., the output of the image conversion unit 41) is input to the first layer 43 of the feature extraction unit. At this time, as shown in the figure, the first layer feature extraction unit coefficient update unit 48 sets the re-learning coefficient for the first layer obtained during re-learning to the first layer 43 of the feature extraction unit. That is, the first layer 43 of the feature extraction unit operates as the re-learned first layer of the re-learned feature extraction unit. Also, the second layer feature extraction unit coefficient update unit sets the re-learning coefficient for the second layer to the second layer of the feature extraction unit. That is, the second layer of the feature extraction unit operates as the second layer of the re-learned feature extraction unit. Therefore, the feature points output by the second layer of the feature extraction unit correspond to the feature points output by the second layer of the re-learned feature extraction unit.

[0072] Similar to the first layer relearning, the difference between the feature points output by the second layer of the feature extraction unit and the feature points stored in the storage unit of the second layer is calculated by the feature extraction unit error calculation unit of the second layer. Furthermore, the feature extraction unit coefficient update unit of the second layer updates and stores the re-learning coefficients for the second layer based on the difference.

[0073] By alternately repeating the above operations, the re-learning coefficients for the second layer are updated so that the feature points output by the second layer of the re-learning feature extraction unit are as close as possible to the feature points output by the second layer of the pre-trained feature extraction unit. This allows the differences between the artificial site image and the simulation image to be further absorbed in the second layer of the re-learning feature extraction unit. In other words, the output of the second layer of the re-learning feature extraction unit becomes feature points that are even closer to the feature points of the simulation image (teacher features (second layer)).

[0074] During (or after) relearning the second layer of the relearning feature extraction unit, relearning of the relearning identification unit 45 is performed again. At that time, of the outputs of the second layer of the feature extraction unit, the output as the second layer of the relearning feature extraction unit is transmitted to the third layer of the feature extraction unit. Also, each layer from the third layer onwards of the feature extraction unit is operated as each layer of the pre-trained feature extraction unit. In other words, the feature extraction unit coefficient update unit of each layer sets pre-trained coefficients in the feature extraction unit of each layer. Re-learning of the relearning identification unit 45 is performed in the same manner as described above.

[0075] After the re-learning in the re-learning identification unit 45 has converged, the accuracy determination unit 414 re-calculates the current recognition accuracy in the same manner as described above. If the calculated current recognition accuracy is equal to or greater than the target value, the re-learning is completed. On the other hand, if the current recognition accuracy is less than the target value, it is determined that the feature points output by the re-learning feature extraction unit still deviate from the teacher features (teacher features (final layer)) with only the first and second layers re-learning. Therefore, the re-learning feature extraction unit continues to re-learn the third and subsequent layers in the same manner, in order, until the recognition accuracy calculated by the accuracy determination unit 414 becomes equal to or greater than the target value. When the recognition accuracy becomes equal to or greater than the target value, the re-learning is completed.

[0076] In the above description, the operating mode of the feature extraction coefficient update unit of each layer, i.e., whether to operate for a pre-trained feature extraction unit or for a re-learned feature extraction unit, is instructed by a mode switching signal from the accuracy determination unit 414.

[0077] As in the first embodiment, after re-learning is completed, inference, i.e., recognition, is performed. At this time, the coefficients of the re-learning feature extraction unit and the re-learning identification unit are fixed to the coefficients obtained during re-learning. Also, for layers of the feature extraction unit that have not been re-learned, the pre-learned coefficients are set as they are. During inference, since the operation of the image conversion unit 41 is not necessary, the operation is turned off to reduce power consumption. Also, as shown in FIG. 2, re-learning and inference may be performed alternately.

[0078] In this embodiment, instead of providing each layer with its own coefficient update unit for the feature extraction unit and / or error calculation unit for the feature extraction unit, each layer may share one or a small number of coefficient update units for the feature extraction unit and / or error calculation unit for the feature extraction unit, since each layer updates its coefficients in different periods.

[0079] Also in this embodiment, instead of comprehensively re-learning the first, second, and third layers of the feature extraction unit from the previous layers, it is possible to re-learn only effective layers from the previous layers, for example, the first, third, and sixth layers. Also, a block such as the feature extraction unit first layer 43 in Fig. 4 may be a block including not only one layer but also multiple layers. Also, as in the first embodiment, the stage following the re-learning feature extraction unit may be a fixed feature extraction unit.

[0080] Thus, in this embodiment, the trained deep neural network and the deep neural network for relearning are configured as a shared deep neural network, and a switching unit 42 is provided that switches between the shared deep neural networks by time-sharing operation.

[0081] The system also includes an error calculation unit for classification unit 412 that calculates the difference between the output data of the re-learning classification unit and the annotation data, and a coefficient update unit for classification unit 413 that updates the coefficients used by the re-learning classification unit based on the difference, and the re-learning classification unit re-learns the classification method for the item based on feature points output by the re-learning deep neural network of the coefficients updated by the coefficient update unit for classification unit.

[0082] As described above, according to this embodiment, it is possible to realize an AI that can recognize on-site images with high accuracy without the need for manual annotation of learning images and can flexibly adapt to changes in the environment and tasks. In addition, since the feature extraction unit is retrained only for the necessary parts, the power consumption, time, and hardware required for retraining can be reduced. Furthermore, the necessary parts can be automatically identified by the accuracy determination unit. In addition, by performing the learning of the feature extraction unit in a time-sharing operation, the time required for learning is doubled compared to the second embodiment, but the hardware required for implementing the feature extraction unit can be halved compared to the second embodiment. EXAMPLES

[0083] The configuration of the fourth embodiment of the present invention is shown in Fig. 6. In this embodiment, instead of using the original image and annotation data generated by simulation in the first embodiment, an actual image (first site real image) of an existing first site and its corresponding annotation data are used. Based on these, the DNN for recognizing items for the second site is retrained.

[0084] The image conversion unit 61 converts the first site real image into the second site artificial image by the configuration and operation of cycle GAN. To learn the conversion, the image conversion unit 61 is also supplied with an actual image of the second site, which is a new site (a learning second site real image).

[0085] The pre-trained feature extraction unit 65 is applied with pre-training using actual images of the first site and the corresponding annotation data. The feature points output by the pre-trained feature extraction unit 65 become teacher features.

[0086] In this manner, in this embodiment, the image conversion unit 61 inputs a first site image (first site real image) which is an existing actual site image and a second site image (learning second site real image) which is a new actual site image into a generative adversarial network and converts the first site image into an image of an artificial site (second site artificial image), the pre-trained feature extraction unit 65 inputs the first site image into a trained deep neural network (e.g., a pre-trained DNN model) trained using the first site image and annotation data for the first site image during re-learning, and outputs feature points of the first site image, and the image of the artificial site is input into a deep neural network for re-learning (e.g., a pre-trained DNN model). The pre-trained feature extraction unit 60 includes a re-learning feature extraction unit 62 that inputs the first site image into a deep neural network (for example, a DNN model having the same network structure as the pre-trained feature extraction unit 15) and re-learns the differences between the first site image and the artificial site image, and outputs feature points of the artificial site image; an error calculation unit 66 for feature extraction unit that calculates the difference between the feature points output by the re-learning feature extraction unit and the feature points output by the pre-trained feature extraction unit; a coefficient update unit 67 for feature extraction unit that updates the coefficients of the re-learning feature extraction unit used for re-learning based on the difference; and a re-learning identification unit 64 that re-learns the method for identifying the item based on the feature points output by the deep neural network for re-learning of the coefficients updated by the coefficient update unit for feature extraction unit.

[0087] This embodiment has the same configuration as the first embodiment and performs the same operation, so detailed description will be omitted. According to this embodiment, by utilizing images and annotation data for existing sites, it is possible to realize an AI that can recognize images of new sites with high accuracy without requiring manual annotation of images of new sites and can flexibly adapt to changes in the environment and tasks. In addition, since re-learning of the feature extraction unit is performed only in the first stage, the power consumption, time, and hardware amount required for re-learning can be reduced. EXAMPLES

[0088] The configuration of the fifth embodiment of the present invention is shown in FIG. 7. In this embodiment, instead of using the original image and annotation data generated by simulation in the second embodiment, an actual image of the first site (first site real image) and the corresponding annotation data are used. Based on these, the DNN for recognizing items for the second site is retrained. The image conversion unit 71 converts the first site real image into the second site artificial image by the configuration and operation of the cycle GAN. To learn the conversion, the image conversion unit 71 is also supplied with an actual image of the second site (second site real image for learning) (not shown).

[0089] Actual images of the first site and their corresponding annotation data are used to apply pre-trained feature extraction unit first layer 75 to pre-trained feature extraction unit final layer 78. The feature points output from pre-trained feature extraction unit first layer 75 to pre-trained feature extraction unit final layer 78 become teacher features (first layer) to teacher features (final layer).

[0090] Thus, in this embodiment, the trained deep neural network is composed of multiple learning layers, the deep neural network for relearning is composed of multiple re-learning layers, the pre-trained feature extraction unit performs the learning in sequence starting from the earlier layers of the multiple learning layers, and the relearning feature extraction unit performs the relearning in sequence starting from the earlier layers of the multiple relearning layers.

[0091] The re-learning feature extraction unit also includes an accuracy determination unit 713 that determines the accuracy of the deep neural network for re-learning, and terminates re-learning based on the accuracy determination result by the accuracy determination unit.

[0092] This embodiment has the same configuration as the second embodiment and performs the same operation, so detailed description will be omitted. According to this embodiment, by utilizing images and annotation data for existing sites, it is possible to realize an AI that can recognize images of new sites with high accuracy without requiring manual annotation of images of new sites and can flexibly adapt to environmental changes and changes in tasks. In addition, since re-learning of the feature extraction unit is performed only on the necessary parts, it is possible to reduce the power consumption, time, and hardware amount required for re-learning. Furthermore, the necessary parts can be automatically identified by the accuracy determination unit. EXAMPLES

[0093] The configuration of the sixth embodiment of the present invention is shown in Fig. 8. In this embodiment, instead of using the original image and annotation data generated by simulation in the third embodiment, an actual image (first site image) of the first site and its corresponding annotation data are used. Based on these, the DNN for recognizing items for the second site is retrained. The image conversion unit 81 converts the first site image into the second site artificial image by the configuration and operation of the cycle GAN. To learn the conversion, the image conversion unit 81 is also supplied with an actual image of the second site (second site image).

[0094] The pre-trained feature extraction unit applies pre-training using actual images of the first site and the corresponding annotation data.

[0095] Thus, in this embodiment, the trained deep neural network and the deep neural network for relearning are configured as a shared deep neural network, and a switching unit 82 is provided that switches between the shared deep neural networks by time-sharing operation.

[0096] This embodiment has the same configuration as the third embodiment and performs the same operation, so detailed description will be omitted. According to this embodiment, by utilizing images and annotation data for existing sites, it is possible to realize an AI that can recognize images of new sites with high accuracy without requiring manual annotation of images of new sites, and can flexibly adapt to environmental changes and changes in tasks. In addition, since the feature extraction unit is re-trained only for the necessary parts, the power consumption, time, and hardware amount required for re-training can be reduced. Furthermore, the necessary parts can be automatically identified by the accuracy determination unit. In addition, by performing the learning of the feature extraction unit in a time-sharing operation, the hardware required for implementing the feature extraction unit can be halved compared to the fifth embodiment. On the other hand, the time required for learning is doubled compared to the fifth embodiment. EXAMPLES

[0097] The configuration of a seventh embodiment of the present invention is shown in Fig. 9. In this embodiment, an example of the operation timing of the image conversion unit is different from that in Fig. 2.

[0098] In this embodiment, the learning and execution of the image conversion unit are not performed simultaneously as in Fig. 2, but are divided into a learning period and an execution period as in Fig. 9. The learning of the image conversion unit is performed during all or part of the period during which the object recognition DNN performs inference.

[0099] During the re-learning period of the DNN for object recognition, the image conversion unit only performs conversion and does not perform learning. Therefore, by turning off the learning function of the cycle GAN, the power consumption of the image conversion unit can be reduced. In addition, when the image conversion unit is neither learning nor performing conversion, the operation of the image conversion unit may be turned off to reduce power consumption.

[0100] According to this embodiment, the overall power consumption during the re-learning period of the DNN for item recognition can be reduced. [Explanation of symbols]

[0101] 11: Image conversion section 12: Retraining feature extraction part 13: Fixed feature extraction unit 14: Re-learning classification unit 15: Pre-trained feature extraction unit 16: Error calculation unit for feature extraction unit 17: Coefficient update unit for feature extraction 18: Error calculation section for identification section 19: Coefficient update unit for the discrimination unit 31: Image conversion section 32: Retraining feature extraction layer 1 33: Final layer of retraining feature extraction 34: Re-learning classification unit 35: Pre-trained feature extraction layer 1 36, 39: Error calculation unit for feature extraction unit 37, 310: Coefficient update unit for feature extraction unit 38: Final layer of pre-trained feature extractor 311: Error calculation section for identification section 312: Coefficient update unit for identification unit 313: Accuracy judgment section 41: Image conversion section 42: Switching section 43: Feature extraction layer 1 44: Final layer of feature extraction 45: Re-learning classification unit 46, 49: Holding part 47, 410: Error calculation unit for feature extraction unit 48, 411: Coefficient update unit for feature extraction unit 412: Error calculation section for identification section 413: Coefficient update unit for identification unit 414: Accuracy judgment section 61: Image conversion section 62: Retraining feature extraction unit 63: Fixed feature extraction unit 64: Re-learning classification unit 65: Pre-trained feature extraction unit 66: Error calculation unit for feature extraction unit 67: Coefficient update unit for feature extraction unit 68: Error calculation section for identification section 69: Coefficient update unit for identification unit 71: Image conversion section 72: Retraining feature extraction layer 1 73: Final layer of retraining feature extraction 74: Re-learning classification unit 75: Pre-trained feature extraction layer 1 76, 79: Error calculation unit for feature extraction unit 77, 710: Coefficient update unit for feature extraction unit 78: Final layer of pre-trained feature extraction 79: Coefficient update unit for identification unit 711: Error calculation section for identification section 712: Coefficient update unit for identification unit 713: Accuracy judgment section 81: Image conversion section 82: Switching section 83: Feature extraction layer 1 84: Final layer of feature extraction 85: Re-learning classification unit 86, 89: Holding part 87, 810: Error calculation unit for feature extraction unit 88, 811: Coefficient update unit for feature extraction unit 812: Error calculation section for identification section 813: Coefficient update unit for identification unit 814: Accuracy judgment section

Claims

1. an image conversion unit that inputs a simulation image and an actual site image into a generative adversarial network and converts the simulation image into an artificial site image that is an image that mimics the site image including an item to be recognized; a pre-trained feature extraction unit that, during re-learning, inputs the simulation image into a trained deep neural network trained using the simulation image and annotation data for the simulation image, and outputs feature points of the simulation image; a re-learning feature extraction unit that inputs the image of the artificial site into a deep neural network for re-learning, re-learns the difference between the image of the artificial site and the simulation image, and outputs feature points of the image of the artificial site; a feature extraction unit error calculation unit that calculates a difference between the feature points output by the re-learned feature extraction unit and the feature points output by the pre-trained feature extraction unit; a feature extraction unit coefficient update unit that updates a coefficient of the re-learning feature extraction unit used for re-learning based on the difference; a re-learning identification unit that re-learns a method for identifying the article based on feature points output by the re-learning deep neural network of the coefficients updated by the feature extraction unit coefficient update unit; A learning recognition device having the above configuration.

2. The trained deep neural network is composed of a plurality of training layers, The deep neural network for relearning is composed of a plurality of relearning layers, the pre-trained feature extraction unit performs the learning in order from a previous layer among the plurality of learning layers, the relearning feature extraction unit performs the relearning in order from a previous layer among the plurality of relearning layers, 2. The learning recognition device according to claim 1.

3. An accuracy determination unit that determines accuracy of the deep neural network for re-learning, the re-learning feature extraction unit terminates re-learning based on a result of the accuracy determination by the accuracy determination unit.

3. The learning recognition device according to claim 2.

4. The trained deep neural network and the re-training deep neural network are configured as a shared deep neural network; A switching unit that switches the shared deep neural network by a time-division operation.

3. The learning recognition device according to claim 2.

5. an error calculation unit for a classification unit that calculates a difference between output data of the re-learning classification unit and the annotation data; a recognition unit coefficient update unit that updates the coefficient used by the re-learning recognition unit based on the difference, the re-learning recognition unit re-learns a method for identifying the object based on feature points output by the re-learning deep neural network of the coefficients updated by the recognition unit coefficient update unit; 2. The learning recognition device according to claim 1.

6. an error calculation unit for a classification unit that calculates a difference between output data of the re-learning classification unit and the annotation data; a recognition unit coefficient update unit that updates the coefficient used by the re-learning recognition unit based on the difference, the re-learning recognition unit re-learns a method for identifying the object based on feature points output by the re-learning deep neural network of the coefficients updated by the recognition unit coefficient update unit; 3. The learning recognition device according to claim 2.

7. an error calculation unit for a classification unit that calculates a difference between output data of the re-learning classification unit and the annotation data; a recognition unit coefficient update unit that updates the coefficient used by the re-learning recognition unit based on the difference, the re-learning recognition unit re-learns a method for identifying the object based on feature points output by the re-learning deep neural network of the coefficients updated by the recognition unit coefficient update unit; 5. The learning recognition device according to claim 4.

8. The image conversion unit, during a period in which the re-learning is not being performed, stopping the conversion of the simulation image into an image of the artificial scene by the 2. The learning recognition device according to claim 1.

9. an image conversion unit that inputs a first site image, which is an existing actual site image, and a second site image, which is a new actual site image, into a generative adversarial network and converts the first site image into an image of an artificial site; a pre-trained feature extraction unit that, during re-learning, inputs the first site image into a trained deep neural network trained using the first site image and annotation data for the first site image, and outputs feature points of the first site image; a re-learning feature extraction unit that inputs the image of the artificial site into a deep neural network for re-learning, re-learns the difference between the image of the artificial site and the first site image, and outputs feature points of the image of the artificial site; a feature extraction unit error calculation unit that calculates a difference between the feature points output by the re-learned feature extraction unit and the feature points output by the pre-trained feature extraction unit; a feature extraction unit coefficient update unit that updates a coefficient of the re-learning feature extraction unit used for re-learning based on the difference; a re-learning identification unit that re-learns an article identification method based on feature points output by the re-learning deep neural network of the coefficients updated by the feature extraction unit coefficient update unit; A learning recognition device having the above configuration.

10. The trained deep neural network is composed of a plurality of training layers, The deep neural network for relearning is composed of a plurality of relearning layers, the pre-trained feature extraction unit performs the learning in order from a previous layer among the plurality of learning layers, the relearning feature extraction unit performs the relearning in order from a previous layer among the plurality of relearning layers, 10. The learning recognition device according to claim 9.

11. An accuracy determination unit that determines accuracy of the deep neural network for re-learning, the re-learning feature extraction unit terminates re-learning based on a result of the accuracy determination by the accuracy determination unit.

11. The learning recognition device according to claim 10.

12. The trained deep neural network and the re-training deep neural network are configured as a shared deep neural network; A switching unit that switches the shared deep neural network by a time-division operation.

11. The learning recognition device according to claim 10.

Images