Control device, method of controlling control device, program, and learning method of neural network

Abstract

To provide a control device capable of suppressing lowering of the accuracy of an inference result.SOLUTION: A control device 100 includes a neural network unit 101 comprising a first inference model 200 and a second inference model 220, a determination unit 102, and a control unit 103, and performs inference processing using related two pieces of input data. The control unit 103 controls the first inference model 200 to output data from a connection layer 210, when the determination unit 102 determines that one of the two pieces of input data cannot be used for the inference processing, on the basis of feature quantities extracted in a layer that uses the data not to be used for the inference processing as input data, out of a first layer 201 and a second layer 202, and feature quantities extracted by the second inference model 220 using the data to be used for the inference processing as input data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a control device, a control method for a control device, a program, and a learning method for a neural network. [Background technology]

[0002] Deep learning technology using neural networks is used in a wide range of fields, and in particular, it is said that the class classification that recognizes and classifies images has exceeded human recognition capabilities. Convolutional Neural Networks (CNNs), which are particularly widely used, achieve high-precision deep learning processing by recursively performing convolution operations on images. Inference models obtained by such deep learning processing have recently been used in facial expression recognition processing that recognizes facial expressions contained in photographed images. In facial expression recognition processing using the above-mentioned inference models, the accuracy of recognizing facial expressions is improved mainly from information such as the unevenness, texture, and contours of the face extracted from the image, but the improvement in accuracy is not sufficient because recognition is performed using only a single modal information such as the photographed image.

[0003] In response to this, a technology has been proposed that performs deep learning processing using multiple modal information. As a related technology, the technology of Patent Document 1 has been proposed. In Patent Document 1, by integrating and learning multiple inference models using multiple modal information, it is possible to improve the accuracy of the inference result compared to the case of learning using single modal information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-2023 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even when the technology of Patent Document 1 mentioned above is used, if any of the multiple modal information is poor modal information, the accuracy of the inference result for the poor modal information decreases, so even if multiple inference models are trained in an integrated manner, the accuracy of the inference result decreases.

[0006] An object of the present invention is to provide a control device, a control method for a control device, a program, and a learning method for a neural network that can suppress a decrease in the accuracy of an inference result. [Means for solving the problem]

[0007] In order to achieve the above object, the control device of the present invention is a control device that performs inference processing using a plurality of related data as input, and includes a first inference model composed of a plurality of layers each receiving the plurality of data as input and extracting features of the input data, and a combined layer that outputs output data that is an inference result based on the extracted features, a second inference model that receives any of the plurality of data as input and extracts features of the input data, control means for controlling the operation of the first inference model and the second inference model, and judgment means for judging whether or not each of the plurality of data can be used in the inference processing, and the control means judges whether all of the plurality of data can be used in the inference processing by the judgment means. When it is determined that the data can be used for the inference process by the determination means, the second inference model does not extract features, and the first inference model controls to output the output data from the combined layer based on the multiple features extracted in the multiple layers, and when it is determined by the determination means that any of the multiple data cannot be used for the inference process, the first inference model controls to output the output data from the combined layer based on the features extracted in a layer of the multiple layers that receives as input data determined by the determination means to be usable for the inference process, and the features extracted by the second inference model that receives as input data determined by the determination means to be usable for the inference process. Effect of the Invention

[0008] According to the present invention, it is possible to suppress a decrease in the accuracy of the inference result. [Brief description of the drawings]

[0009] [Figure 1] 2 is a block diagram showing an example of the configuration of a control device according to the present embodiment; FIG. [Diagram 2] 2 is a block diagram showing an example of the configuration of a neural network unit in FIG. 1. [Diagram 3] 2 is a diagram showing an example of image data used in the control device of FIG. 1; [Figure 4] 2 is a diagram showing an example of sound data used in the control device of FIG. 1; [Diagram 5] 2 is a diagram showing an example of sound data used in the control device of FIG. 1; [Figure 6] 4 is a flowchart showing the procedure of an attention area estimation process executed by the control device of FIG. [Figure 7] FIG. 2 is a diagram illustrating a method for learning parameters used in the neural network unit of FIG. 1. [Figure 8] 2 is a diagram showing an example of image data used in the control device of FIG. 1; [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of a control device that receives three related input data. [Figure 10] 10 is a block diagram showing an example of the configuration of the neural network unit in FIG. 9. [Figure 11] FIG. 4 is a diagram showing an example of image data generated by continuous shooting. [Figure 12] 10 is a flowchart showing the procedure of image quality improvement processing executed by the control device of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] 1 is a block diagram showing an example of the configuration of a control device 100 according to this embodiment. The control device 100 performs an integrated inference process based on two related input data, and outputs an inference result. In this embodiment, as an example, a configuration will be described in which the control device 100 performs an inference process to infer an attention area in image data based on image data input as input data 1 and sound data input as input data 2. The attention area is an area focusing on people having a conversation in the image data.

[0012] In FIG. 1, the control device 100 includes a neural network unit 101, a determination unit 102, and a control unit 103. The neural network unit 101 receives image data and sound data as input, performs inference processing of the region of interest according to instructions from the control unit 103, and outputs the inference result as output data. The output data is output to, for example, an internal storage (not shown) included in the control device 100, or an external device that can communicate with the control device 100 via a communication network such as the Internet. In this embodiment, the neural network unit 101 is a processing unit capable of product-sum calculation and nonlinear processing adopted in a general neural network model. The determination unit 102 determines whether the input image data and sound data can be used for inference processing of the region of interest, and transmits the determination result to the control unit 103. The determination method will be described later. The control unit 103 transmits an instruction to the neural network unit 101 to control the inference processing of the region of interest based on the determination result acquired from the determination unit 102.

[0013] Fig. 2 is a block diagram showing an example of the configuration of the neural network unit 101 in Fig. 1. In this embodiment, the neural network unit 101 is configured with a convolutional neural network (CNN). Generally, CNN processing also includes bias addition and nonlinear processing, but details thereof are omitted in this embodiment. However, the present invention does not limit the configuration of the neural network unit 101 to only CNN, and may include calculations such as fully connected processing.

[0014] In FIG. 2, the neural network unit 101 is made up of a first inference model 200 and a second inference model 220.

[0015] The first inference model 200 is composed of a first layer 201, a second layer 202, and a combined layer 210.

[0016] The first layer 201 receives image data as input and extracts features of a person of interest from the image data. The first layer 201 extracts the feature of the person of interest, for example, the likelihood of the person being a person of interest based on three aspects: the right eye, the left eye, and the nose.

[0017] The second layer 202 receives sound data as input and extracts features of the person of interest from the sound data. For example, the second layer 202 extracts the likelihood of the person of interest being a person of interest based on the pitch of the sound as the feature of the person of interest.

[0018] The second inference model 220 is a model that has been pre-trained so that the accuracy of the inference result does not decrease even if input data determined to be unusable by the judgment unit 102 is not used, and is composed of a fourth layer 221 and a fifth layer 222.

[0019] The fourth layer 221, like the first layer 201, receives image data as input and extracts features of the person of interest from the image data. The fourth layer 221 extracts the likelihood of the person of interest being a person of interest from a perspective different from that of the first layer 201, for example, the perspective of the mouth, as the feature of the person of interest. In this way, in this embodiment, the fourth layer 221 extracts the likelihood of the person of interest being a person of interest from a perspective different from that of the first layer 201.

[0020] The fifth layer 222, like the second layer 202, receives sound data as input and extracts features of the person of interest from the sound data. The fifth layer 222 extracts the likelihood of the person of interest as a person of interest from a perspective different from that of the second layer 202, for example, the perspective of volume, as the feature of the person of interest. Thus, in this embodiment, the fifth layer 222 extracts the likelihood of the person of interest as a person of interest from a perspective different from that of the second layer 202.

[0021] The combining layer 210 performs an inference process to combine the feature quantities extracted by two layers selected from the first layer 201, the second layer 202, the fourth layer 221, and the fifth layer 222 based on the result of the determination made by the determination unit 102 as to whether the input data can be used. The combining layer 210 outputs the inference result as output data. Note that, in the layers other than the two layers selected from the first layer 201, the second layer 202, the fourth layer 221, and the fifth layer 222, no calculation is performed to extract the feature quantities. Therefore, a predetermined fixed value is input to the combining layer 210 instead of the feature quantities as the output of the layer in which the calculation is not performed.

[0022] Next, a description will be given of the determination as to whether or not the input data can be used, performed by the determination unit 102. First, a description will be given of the determination as to whether or not the image data, which is the input data, can be used.

[0023] FIG. 3 is a diagram showing an example of image data used in the control device 100 of FIG.

[0024] The image data of Fig. 3(a) and the image data of Fig. 3(b) respectively include a person 300 having a conversation and a person 301 walking. Note that the image data of Fig. 3(b) has a lower brightness than the image data of Fig. 3(a).

[0025] The determination unit 102 calculates an evaluation value of the input image data. The determination unit 102 calculates, for example, an average luminance value as the evaluation value of the input image data. If the calculated average luminance value exceeds a predetermined luminance threshold value, the determination unit 102 determines that the input image data is usable. For example, the image data of FIG. 3(a) in which the calculated average luminance value exceeds the luminance threshold value is determined to be usable by the determination unit 102. In this embodiment, the luminance threshold value is set to a luminance level value at which human features can be extracted in the first layer 201.

[0026] On the other hand, when the calculated average value of luminance is equal to or less than the luminance threshold value, the determination unit 102 determines that the input image data is unusable. For example, the image data in FIG. 3(b) in which the calculated average value of luminance is equal to or less than the luminance threshold value is determined to be unusable by the determination unit 102. In the present embodiment, as an example, a configuration is described in which the usability of the image data is determined based on the average value of luminance of the input image data, but the present invention is not limited to this configuration. For example, the usability of the image data may be determined based on the SN ratio of the luminance of the input image data. In addition, the usability of the image data may be determined based on parameters of the image data that can be compared with a predetermined threshold value, such as the amount of blur and the amount of high sensitivity noise in the input image data. In addition, the usability of the image data may be determined by combining the results of comparing these multiple parameters in the input image data with the threshold values ​​corresponding to each parameter.

[0027] Next, a description will be given of a determination as to whether or not sound data, which is input data, can be used.

[0028] Fig. 4 is a diagram showing an example of sound data used in the control device 100 of Fig. 1. Fig. 4(a) shows an example of a waveform of sound data input to the control device 100. This sound data includes voice components of a person 300 having a conversation and sound components generated in the surrounding environment.

[0029] In this embodiment, when sound data of Fig. 4(a) is input, for example, the determination unit 102 separates this sound data into a waveform of Fig. 4(b) and a waveform of Fig. 4(c). Fig. 4(b) is a waveform obtained by separating the voice components of a person 300 having a conversation from the waveform of Fig. 4(a). Fig. 4(c) is a waveform obtained by separating noise components other than the voice components (characteristic components) from the waveform of Fig. 4(a). In this embodiment, the waveform of Fig. 4(c) is used as an evaluation value of the input sound data.

[0030] Next, when the maximum amplitude of the noise component in Fig. 4(c) is equal to or less than a predetermined noise threshold, the determination unit 102 determines that the sound data shown in Fig. 4(a) is usable. Note that the noise threshold is set to, for example, the maximum amplitude of the voice component of the person 300 having a conversation in Fig. 4(b).

[0031] Fig. 5 is also a diagram showing an example of sound data used in the control device 100 of Fig. 1. Fig. 5(a) shows an example of a waveform of sound data having a larger noise component than the sound data of Fig. 4(a).

[0032] For example, when the sound data of FIG. 5(a) is input, the determination unit 102 separates the sound data into the waveform of FIG. 5(b) and the waveform of FIG. 5(c). FIG. 5(b) is a waveform obtained by separating the voice components of a person 300 who is talking from the waveform of FIG. 5(a). Note that the voice components in the sound data of FIG. 5(a) are the same as the voice components in the sound data of FIG. 4(a), and the waveform of FIG. 5(b) is equivalent to the waveform of FIG. 4(b). FIG. 5(c) is a waveform obtained by separating noise components other than the voice components from the waveform of FIG. 5(a), and the noise components are larger than those in the waveform of FIG. 4(c). The waveform of FIG. 5(c) is also used as an evaluation value for the input sound data.

[0033] When the maximum value of the amplitude of the noise component in Fig. 5(c) exceeds a predetermined noise threshold, the determination unit 102 determines that the sound data shown in Fig. 5(a) is unusable. Note that, in the present embodiment, a configuration has been described in which the usability of the input sound data is determined based on the noise component of the sound data, but the present invention is not limited to this configuration. For example, the usability of the sound data may be determined based on a parameter of the sound data that can be compared with a predetermined threshold, such as the volume of the input sound data.

[0034] Here, while only the first layer 201 that receives image data as input can recognize a person from the image data, it is difficult to identify whether the person is a talking person 300 or a walking person 301. In contrast, this embodiment is configured to be able to identify the talking person 300 by performing integrated inference processing using the second layer 202 that receives sound data as input and the first layer 201 that receives image data as input.

[0035] On the other hand, if the input sound data is defective data that is determined by the determination unit 102 to be unusable for the inference process, there is a concern that the accuracy of the inference result will decrease if the inference process is performed using this sound data. For this reason, it is preferable to perform the inference process without using this sound data. However, as described above, if the inference process is performed using only the first layer 201 without using this sound data, it is not possible to accurately identify whether the person recognized from the image data is a person 300 who is talking or a person 301 who is walking.

[0036] In contrast, in this embodiment, when the judgment unit 102 judges that either the input image data or the input sound data cannot be used for the inference process, the following process is performed. That is, output data is output from the combination layer 210 based on the feature amount extracted in the layer that receives the data judged to be usable for the inference process out of the first layer 201 and the second layer 202 as input, and the feature amount extracted by the second inference model 220 using the data as input.

[0037] Fig. 6 is a flowchart showing the procedure of an attention area inference process executed by the control device 100 of Fig. 1. The attention area inference process of Fig. 6 is executed when the control device 100 acquires image data and sound data as input data.

[0038] In FIG. 6, first, in S601, the determination unit 102 determines whether all the acquired input data can be used for the inference process. Specifically, the determination unit 102 calculates an evaluation value for each acquired input data, and compares the calculated evaluation value with a predetermined threshold to determine whether each input data can be used for the inference process. For example, as described above, the determination unit 102 calculates an average value of luminance of the image data that is the input data, and compares the calculated average value of luminance with the luminance threshold. Also, as described above, the determination unit 102 extracts noise components of the sound data that is the input data, and compares the maximum amplitude of the noise components with the noise threshold. For example, when the image data of FIG. 3(a) in which the average value of luminance exceeds the luminance threshold and the sound data of FIG. 4(a) in which the maximum amplitude of the noise components is equal to or less than the noise threshold are input, in S601, it is determined that all the acquired input data can be used. In this case, the inference process of the attention area proceeds to S602.

[0039] In S602, the control unit 103 controls the first inference model 200 to input the output of the first layer 201 and the output of the second layer 202 to the combined layer 210. The control unit 103 also controls the second inference model 220 not to perform the calculations of the fourth layer 221 and the fifth layer 222. Thus, in this embodiment, when it is determined that both the acquired image data and sound data are usable, the control unit 103 controls the second inference model 220 not to extract features. The control unit 103 also controls the first inference model 200 to output output data from the combined layer 210 based on the multiple features extracted in the first layer 201 and the second layer 202.

[0040] Next, in S603, the control unit 103 controls the first inference model 200 and the second inference model 220 so as to input a predetermined fixed value to the coupling layer 210 instead of the output of the layer that does not perform the calculation. Specifically, the control unit 103 controls the second inference model 220 so as to input a fixed value corresponding to the fourth layer 221 and a fixed value corresponding to the fifth layer 222 to the coupling layer 210, respectively, instead of the output of the fourth layer 221 and the output of the fifth layer 222 that do not perform the calculation. The predetermined fixed value is a value used when the first inference model 200 and the second inference model 220 are trained. By controlling in this way, the neural network unit 101 composed of the first inference model 200 and the second inference model 220 can determine that the processing does not perform the calculation of the second inference model 220. As a result, it is possible to output an inference result with higher accuracy than when data not linked to the learning is input to the coupling layer 210.

[0041] Next, in S604, the neural network unit 101 performs neural network calculations under the control of the control unit 103, and outputs the inference result as output data. After that, the inference process of the attention area is completed. In this manner, in this embodiment, the area that focuses on the person 300 who is talking, such as the area 302 in FIG. 3(c), can be specified as the attention area.

[0042] On the other hand, if it is determined in S601 that any of the acquired input data is unusable, the inference process of the attention area proceeds to S605. For example, if the image data of FIG. 3(b) in which the average value of luminance is equal to or less than the luminance threshold value is input, or if the sound data of FIG. 5(a) in which the maximum amplitude of the noise component exceeds the noise threshold value is input, it is determined in S601 that any of the acquired input data is usable. In this embodiment, if image data in which the average value of luminance is equal to or less than the luminance threshold value and sound data in which the maximum amplitude of the noise component exceeds the noise threshold value are input, it is determined as follows. That is, among these input data, the input data in which the difference between the evaluation value and the threshold value is small is determined to be usable, and the input data in which the difference between the evaluation value and the threshold value is large is determined to be unusable.

[0043] In S605, the control unit 103 controls the first inference model 200 and the second inference model 220 to perform calculations of layers that use input data determined to be unusable as input, and to perform calculations of layers that use input data determined to be usable as input.

[0044] For example, when the determination unit 102 determines that the image data is usable and that the sound data is unusable, the image data is good quality data and the sound data is bad quality data. In such a case, the control unit 103 controls the first inference model 200 and the second inference model 220 so as not to perform the calculations of the second layer 202 and the fifth layer 222 using the sound data determined to be unusable as input. The control unit 103 also controls the first inference model 200 and the second inference model 220 so as to perform the calculations of the first layer 201 and the fourth layer 221 using the image data determined to be usable as input. By controlling in this way, the resources used for the image data, which is good quality data, can be increased by the resources of the second inference model 220. Furthermore, in the second inference model 220, the feature amount of the person of interest based on a different viewpoint from that of the first layer 201 in the first inference model 200 is extracted, so that it is possible to suppress the deterioration of the accuracy of the inference result of the region of interest based on this feature amount without using the sound data, which is bad quality data.

[0045] Also, when the determination unit 102 determines that the image data is unusable and that the sound data is usable, the image data is bad data and the sound data is good data. In such a case, the control unit 103 controls the first inference model 200 and the second inference model 220 so as not to perform the calculations of the first layer 201 and the fourth layer 221 using the image data determined to be unusable as input. Also, the control unit 103 controls the first inference model 200 and the second inference model 220 so as to perform the calculations of the second layer 202 and the fifth layer 222 using the sound data determined to be usable as input. By controlling in this way, the resources used for the sound data, which is good data, can be increased by the resources of the second inference model 220. Also, in the second inference model 220, the feature amount of the person of interest based on a different viewpoint from that of the second layer 202 in the first inference model 200 is extracted, so that it is possible to suppress a decrease in the accuracy of the inference result of the attention area based on this feature amount without using the image data, which is bad data.

[0046] Next, the inference process of the region of interest proceeds to S603. For example, if the determination unit 102 determines that the image data is usable and that the sound data is unusable, in S603, the control unit 103 controls the first inference model 200 and the second inference model 220 as follows. That is, the control unit 103 controls the first inference model 200 and the second inference model 220 to be input to the combined layer 210, instead of the output of the second layer 202 and the output of the fifth layer 222, which do not perform calculations. Also, if the determination unit 102 determines that the image data is unusable and that the sound data is usable, in S603, the control unit 103 controls the first inference model 200 and the second inference model 220 as follows. That is, the control unit 103 controls so that a fixed value corresponding to the first layer 201 and a fixed value corresponding to the fourth layer 221 are input to the coupling layer 210, respectively, instead of the output of the first layer 201 and the output of the fourth layer 221, which do not perform calculations. Note that these fixed values ​​are also values ​​used when the first inference model 200 and the second inference model 220 were learned. By controlling in this way, the neural network unit 101 composed of the first inference model 200 and the second inference model 220 can determine that the processing is performed using only one of the image data and the sound data. As a result, it is possible to output an inference result with higher accuracy than when data not linked to learning is input to the coupling layer 210. Next, the inference processing of the attention area proceeds to S604.

[0047] According to the above-described embodiment, when the determination unit 102 determines that any of the input data cannot be used for the inference process, the following process is performed. That is, output data is output from the combined layer 210 based on the feature amount extracted in the layer that receives the data determined to be usable for the inference process from among the first layer 201 and the second layer 202 as input, and the feature amount extracted by the second inference model 220 using the data as input. In this way, in addition to the feature amount extracted in the layer that receives the data determined to be usable for the inference process from among the first layer 201 and the second layer 202 as input, the inference process can be performed using the feature amount extracted by the second inference model 220 from a different perspective from the layer. This makes it possible to suppress a decrease in the accuracy of the inference result.

[0048] In the above-described embodiment, an evaluation value is calculated for each piece of input data, and whether or not the input data can be used in the inference process is determined based on the result of comparing the calculated evaluation value with a threshold value. This makes it possible to prevent a decrease in the accuracy of the inference result even if any of the input data cannot be used in the inference process.

[0049] In the above-described embodiment, the input data is image data and sound data, so that in the inference process in which image data and sound data are input, it is possible to suppress a decrease in the accuracy of the inference result.

[0050] In the above-described embodiment, the evaluation value is a value related to any one of the luminance of the image data, the amount of blur of the image data, and the amount of high-sensitivity noise of the image data. This makes it possible to suppress a decrease in the accuracy of the inference result even if poor image data caused by the luminance, the amount of blur, or the amount of high-sensitivity noise is input.

[0051] In the above-described embodiment, the value relates to the amount of noise for the characteristic components of the sound data, so that even if poor sound data caused by noise is input, a decrease in the accuracy of the inference result can be suppressed.

[0052] Next, learning of parameters used in the neural network unit 101 will be described.

[0053] Fig. 7 is a diagram illustrating a method for learning parameters used in neural network unit 101 in Fig. 1. In this embodiment, the parameters used in neural network unit 101 are learned in advance using a separate device such as a PC. In this embodiment, learning is performed three times while changing the configuration of neural network unit 101.

[0054] FIG. 7(a) is a diagram for explaining the first learning.

[0055] In the neural network processing in FIG. 7(a), learning is performed using a model configuration in which, among the configurations shown in FIG. 2, calculations are performed in the first layer 201 and the second layer 202, calculations are not performed in the fourth layer 221 and the fifth layer 222, and predetermined fixed values ​​are input to the coupling layer 210 instead of the output of the fourth layer 221 and the output of the fifth layer 222. In this embodiment, the predetermined fixed value is set to a zero value having the same data length as the input data of the coupling layer 210. In this embodiment, the fixed value is set to a zero value as an example, but the fixed value is not limited to a zero value. In this embodiment, the same value as the fixed value used during learning, for example, a zero value, is used as the fixed value in S603. By using the same value as the fixed value used during learning as the fixed value in S603 during inference, the neural network unit 101 can determine that the processing does not involve calculation of the second inference model 220.

[0056] In the first learning (first learning), a plurality of image data and a plurality of sound data are used as input data. Note that the image data and the sound data are related data. Note that only good quality data that is determined to be usable by the determination unit 102 is used in the first learning data. By using only good quality data, it is possible to appropriately perform learning that corresponds to the case where good quality image data and good quality sound data are input. The first learning is performed using the above model configuration and learning data.

[0057] The first learning round is performed by obtaining updated parameters processed by parameter optimization processing so that the relationship between the output data, which is the calculation result of the neural network processing, and the teacher data is closer, and updating the dictionary data of the neural network.

[0058] The updated parameters obtained at the completion of the first learning are used as pre-trained dictionary data for the second learning. Note that the training data can be generated in advance using image data and audio data as inputs to generate data of the area of ​​interest, and the same training data can be used for the first, second, and third learnings. Data generated using the pre-trained dictionary data can also be used as training data. The pre-trained dictionary data can also be set to initial values ​​and used for the second learning.

[0059] Next, the second learning will be described. Fig. 7(b) is a diagram for explaining the second learning.

[0060] In the neural network processing in Figure 7(b), learning is performed using a model configuration in which, of the configuration shown in Figure 2, calculations are performed in the first layer 201 and the fourth layer 221, no calculations are performed in the second layer 202 and the fifth layer 222, and predetermined fixed values ​​are input to the coupling layer 210 instead of the output of the second layer 202 and the output of the fifth layer 222.

[0061] In the second learning, a plurality of image data are used as input data. Note that, in the second learning data, only good quality image data that is determined as usable by the determination unit 102 is used. The second learning is performed using the above model configuration and learning data. Note that, the second learning is performed in the same manner as the first learning. The update parameters obtained at the time of completion of the second learning are used as pre-learning dictionary data for the third learning.

[0062] Next, the third learning (second learning) will be described. Fig. 7(c) is a diagram for explaining the third learning.

[0063] In the neural network processing in Figure 7(c), learning is performed using a model configuration in which, of the configuration shown in Figure 2, calculations are performed in the second layer 202 and the fifth layer 222, no calculations are performed in the first layer 201 and the fourth layer 221, and predetermined fixed values ​​are input to the coupling layer 210 instead of the output of the first layer 201 and the output of the fourth layer 221.

[0064] In the third learning, a plurality of pieces of sound data are used as input data. Note that, in the third learning data, only good quality sound data that is determined to be usable by the determination unit 102 is used. The third learning is performed using the above model configuration and learning data. Note that, the third learning is performed in the same manner as the first learning.

[0065] In this embodiment, a configuration using two types of input data has been described, but it is also possible to use three or more types of input data and perform learning using a model configuration having three or more processing layers corresponding to those data, and furthermore, learning may be performed multiple times.

[0066] The above is an overview of the learning. Note that learning algorithms for neural networks, including the backpropagation method, are within the scope of known techniques and therefore will not be described in this embodiment.

[0067] In this manner, in this embodiment, the first inference model 200 and the second inference model 220 used in the inference processing of the region of interest in Figure 6 described above can be obtained, thereby suppressing a decrease in the accuracy of the inference results.

[0068] In this embodiment, the neural network unit 101 may be configured to perform an inference process for identifying a person appearing in a plurality of image data obtained by photographing a person as a subject from different angles, using the plurality of image data obtained by photographing a person from different angles as input. This makes it possible to suppress a decrease in the accuracy of the inference result in the inference process for identifying information on a specific person appearing in the image data obtained by inputting the plurality of image data obtained from different angles.

[0069] In such a configuration, the neural network unit 101 receives two sets of image data captured from different angles as input, and outputs information on a specific person captured in these sets of image data as output data. For example, the neural network unit 101 receives the image data of FIG. 8(a) obtained by capturing a subject from the front and the image data of FIG. 8(b) obtained by capturing a subject from directly behind as input, and outputs information on the specific person from these sets of image data as output data. Note that the information on the specific person may be in the form of image data or text data, as long as it is information that can identify a person.

[0070] In such a configuration, the determination unit 102 determines whether the input data can be used based on whether the person in the image data can be determined to be a specific person. In this embodiment, the determination unit 102 determines whether the person in the image data can be determined to be a specific person based on the direction of the person's face. For example, as shown in FIG. 8(a), image data obtained by photographing a subject from the front contains information that can identify a person, such as the eyes, nose, and mouth, and the like, so the determination unit 102 determines that such image data can be used. On the other hand, as shown in FIG. 8(b), image data obtained by photographing a subject from directly behind does not contain information that can identify a person, such as the eyes, nose, and mouth, and the like, so the determination unit 102 determines that such image data cannot be used.

[0071] In this way, by the judgment unit 102 determining whether or not the input data can be used based on the direction of the person's face, in the inference process of identifying information about a specific person appearing in two input image data, even if one of the two image data does not contain information that can identify a person, as in Figure 8(b), deterioration in the accuracy of the inference result can be suppressed.

[0072] In addition, in this embodiment, the determination unit 102 may determine whether image data as input data can be used based on the shooting conditions of the image data. The shooting conditions are information at the time of shooting the image data, such as the name of the shooting device, resolution, shutter speed, aperture (F value), ISO sensitivity, photometry mode, whether or not a flash was used, exposure correction step value, and focal length. For example, if the ISO value of the image data is significantly higher than the regular ISO of the camera that took the image, there is a concern that the accuracy of the inference result will deteriorate depending on the degree of deterioration of the image quality. Therefore, if the ISO value of the image data is higher than the range of regular ISO, the determination unit 102 determines that the image data cannot be used. This makes it possible to suppress deterioration of the accuracy of the inference result due to the shooting conditions of the input image data.

[0073] In addition, in this embodiment, the control device may be configured to receive three or more related input data. In the following, as an example, a configuration in which the control device receives three related input data will be described.

[0074] 9 is a block diagram showing an example of the configuration of a control device 900 that receives three pieces of related input data. The control device 900 performs integrated inference processing based on the three pieces of related input data, and outputs an inference result.

[0075] In FIG. 9, the control device 900 includes a neural network unit 901, a determination unit 902, and a control unit 903. The neural network unit 901 receives three pieces of image data in a time series, performs inference processing, specifically, image quality improvement processing described later, according to instructions from the control unit 903, and outputs image data that has been image quality improved as output data. The output data is output to, for example, an internal storage (not shown) included in the control device 900, or an external device that can communicate with the control device 900 via a communication network such as the Internet. The neural network unit 901 is also a processing unit capable of product-sum calculation and nonlinear processing adopted in a general neural network model, similar to the above-mentioned neural network unit 101. The determination unit 902 determines whether the input image data can be used for inference processing or not, and transmits the determination result to the control unit 903. The determination method will be described later. The control unit 903 transmits an instruction to the neural network unit 901 to control the image quality improvement processing based on the determination result acquired from the determination unit 902.

[0076] FIG. 10 is a block diagram showing an example of the configuration of the neural network unit 901 in FIG. 9. The neural network unit 901 receives three pieces of image data in a time series, specifically, image data of the image data of the frames before and after the image data of interest, and performs image quality improvement processing according to an instruction from the control unit 903. In the image quality improvement processing, noise reduction is performed to remove noise from the image data of interest using image data of the frames before and after the image data of interest as reference image data. In this way, by performing noise reduction using multiple image data, the detection accuracy of edge parts and noise parts in the image data is improved, and highly accurate noise removal can be realized. In this embodiment, as an example, image data of a frame before the image data of interest is input as input data 1, the image data of interest is input as input data 2, and image data of a frame after the image data of interest is input as input data 3.

[0077] In FIG. 10, the neural network unit 901 is composed of a first inference model 1000 and a second inference model 1020.

[0078] The first inference model 1000 is composed of a first layer 1001 that receives input data 1 as input, a second layer 1002 that receives input data 2 as input, a third layer 1003 that receives input data 3 as input, and a combination layer 1010.

[0079] The first layer 1001, the second layer 1002, and the third layer 1003 are neural network layers for performing noise reduction, and extract features of input image data. For example, information indicating high-frequency edges and information indicating spatially peculiar pixels are extracted as features of image data.

[0080] The second inference model 1020 is a model that has been trained in advance so that the accuracy of the inference result does not decrease even if the input data determined by the determination unit 902 to be unusable is not used. The second inference model 1020 is composed of a fourth layer 1021 that receives input data 1 as an input, a fifth layer 1022 that receives input data 2 as an input, and a sixth layer 1023 that receives input data 3 as an input. The fourth layer 1021, the fifth layer 1022, and the sixth layer 1023 extract features from a different perspective than the first layer 1001, the second layer 1002, and the third layer 1003, respectively. For example, the fourth layer 1021, the fifth layer 1022, and the sixth layer 1023 each extract information on a human skin region as a feature of the input image data. It should be noted that the processing time would be long if the first layer 1001, the second layer 1002, and the third layer 1003 were to process the fourth layer 1021, the fifth layer 1022, and the sixth layer 1023, respectively. For this reason, this embodiment is configured so that the first layer 1001, the second layer 1002, and the third layer 1003 perform the processes that have a significant effect on image quality.

[0081] In this embodiment, the computational resources used for the fourth layer 1021, the fifth layer 1022, and the sixth layer 1023 are changeable, and the control unit 903 determines these computational resources based on the result of the judgment by the judgment unit 902 as to whether or not the input data can be used.

[0082] The combined layer 1010 outputs output data based on the output of a layer that receives input data determined to be usable by the determination unit 902 among the first layer 1001, the second layer 1002, the third layer 1003, the fourth layer 1021, the fifth layer 1022, and the sixth layer 1023. For example, the combined layer 1010 identifies noise in the image data of interest based on the feature amount acquired from the layer that receives input data determined to be usable by the determination unit 902, and performs an averaging process on the identified noise to remove noise. The combined layer 1010 also retains the edge portion in the image data of interest based on edge information, which is the acquired feature amount. In this way, image data that has been subjected to high image quality processing and from which noise has been removed is output as output data.

[0083] Next, the judgment of the judgment unit 902 as to whether input data can be used will be described. The judgment unit 902 calculates the luminance difference between the inputted image data of interest and the reference image data, and judges whether the inputted image data can be used based on the result of comparing the luminance difference with a threshold value. The threshold value is, for example, a value determined based on the luminance difference of a plurality of related image data used as learning data. The luminance difference between the image data of interest and the reference image data may be calculated based on the average or variance of a certain region. At this time, the luminance difference between the image data may be calculated after predicting the movement between the image data and performing position alignment. The judgment unit 902 judges that image data whose calculated luminance difference is equal to or less than the threshold value can be used.

[0084] FIG. 11 is a diagram showing an example of image data generated by continuous shooting. FIG. 11(a) is image data of a frame before the image data of interest in FIG. 11(b). FIG. 11(b) is the image data of interest. FIG. 11(c) is image data of a frame after the image data of interest in FIG. 11(b). FIG. 11(d) is image data of a frame after the image data of interest in FIG. 11(b), and is image data with a higher luminance value than the image data in FIG. 11(c) due to the effect of flash emission. It is assumed that the position of the person's face is the same in all of the image data in FIG. 11(a) to FIG. 11(d).

[0085] For example, when the target image data in Fig. 11(b) and the reference image data in Fig. 11(a) or 11(c) in which the luminance difference between the target image data and the reference image data is equal to or less than a threshold value are input, it is possible to output image data that has been subjected to high image quality processing in which noise has been removed using these reference image data. Therefore, when the reference image data in which the luminance difference between the target image data and the reference image data is equal to or less than a threshold value is input, the determination unit 902 determines that the input image data is usable.

[0086] On the other hand, when the reference image data of FIG. 11(d) in which the luminance difference between the target image data of FIG. 11(b) and the target image data exceeds the threshold is input, if the image quality improvement process is performed using this reference image data, the noise part cannot be properly identified and the edge part cannot be properly detected. As a result, it is not possible to output the image quality improvement process-processed image data in which the noise has been removed. Therefore, when the reference image data in which the luminance difference between the target image data and the target image data exceeds the threshold is input, the determination unit 902 determines that the input image data cannot be used.

[0087] Fig. 12 is a flowchart showing the procedure of the image quality improvement process executed by the control device 900 of Fig. 9. The image quality improvement process of Fig. 9 is executed when the control device 100 acquires three pieces of image data obtained by continuous shooting as input data.

[0088] In FIG. 12, first, in S1201, the determination unit 902 determines whether all acquired input data are usable. Specifically, the determination unit 902 calculates an evaluation value for each acquired input data, and compares the calculated evaluation value with a predetermined threshold to determine whether each input data is usable for inference processing. For example, as described above, the determination unit 902 calculates the luminance difference between the acquired image data of interest and reference image data, and determines whether the acquired input data is usable based on the result of comparing the luminance difference with the threshold. For example, when the reference image data of FIG. 11(a), the image data of interest of FIG. 11(b), and the reference image data of FIG. 11(c) are input as input data 1, input data 2, and input data 3, respectively, the luminance difference between these reference image data and image data of interest is equal to or less than the threshold. In such a case, in S1201, it is determined that all acquired input data are usable. In this case, the image quality improvement process proceeds to S1202.

[0089] In S1202, the control unit 903 controls the first inference model 1000 so as to input the output of the first layer 1001, the output of the second layer 1002, and the output of the third layer 1003 to the combined layer 1010. The control unit 903 also controls the second inference model 1020 so as not to perform the calculations of the fourth layer 1021, the fifth layer 1022, and the sixth layer 1023. Thus, in this embodiment, when it is determined that all the acquired input data is usable, the control unit 903 controls as follows. That is, the control unit 903 controls the second inference model 1020 not to extract features, and the first inference model 1000 to output output data from the combined layer 1010 based on the multiple features extracted in the first layer 1001, the second layer 1002, and the third layer 1003.

[0090] Next, in S1203, the control unit 903 controls the first inference model 1000 and the second inference model 1020 to input predetermined fixed values ​​to the coupling layer 1010 instead of the output of the layer that does not perform calculations. Specifically, the control unit 903 controls the second inference model 1020 to input fixed values ​​corresponding to the fourth layer 1021, the fifth layer 1022, and the sixth layer 1023 to the coupling layer 1010 instead of the output of the fourth layer 1021, the output of the fifth layer 1022, and the output of the sixth layer 1023 that do not perform calculations. Note that the predetermined fixed values ​​are values ​​used when the first inference model 1000 and the second inference model 1020 are trained.

[0091] Next, in S1204, the neural network unit 901 performs neural network calculations in accordance with the control of the control unit 903, and outputs the image data that has been subjected to image quality improvement processing as output data. After that, the image quality improvement processing ends.

[0092] On the other hand, for example, when the reference image data of FIG. 11(a), the target image data of FIG. 11(b), and the reference image data of FIG. 11(d) are input as input data 1, input data 2, and input data 3, respectively, the following determination is made. That is, the reference image data of FIG. 11(d) (input data 3) whose luminance difference with the target image data exceeds the threshold value is determined to be unusable. In such a case, in S1201, it is determined that any of the acquired input data is unusable. In this case, the control unit 903 weights each of the acquired image data based on the luminance difference with respect to the target image data. For example, when the luminance differences with respect to the target image data of FIG. 11(b) are 2, 1, and 10 in the acquired image data of FIG. 11(a), FIG. 11(b), and FIG. 11(d), respectively, the determination unit 902 sets weighting values ​​of 1, 2, and 0 for these image data.

[0093] Next, in S1205, the control unit 903 determines the computation resources to be used for the fourth layer 1021 and the fifth layer 1022, which are layers that receive data determined to be usable by the determination unit 902 as input, based on the set weighting values. For example, as described above, when the weighting values ​​of the image data of each of Figs. 11(a), 11(b), and 11(d) are 1, 2, and 0, the control unit 903 controls the computation resources of the second inference model 1020 to be divided in a ratio of 1:2:0. By controlling in this way, it is possible to allocate more computation resources to the fourth layer 1021 that extracts features from the reference image data, thereby improving the computation accuracy of the fourth layer 1021. As a result, it is possible to improve the accuracy of the inference result in the image quality improvement process. Note that the computation resources of the second inference model 1020 may be set to be equal to the computation resources of the first layer 1001 that receives unusable data as input. In this way, it is possible to suppress a decrease in the accuracy of the inference result without changing the amount of computation resources for performing the inference process.

[0094] Next, in S1206, the control unit 903 controls the first inference model 1000 and the second inference model 1020 to perform calculations on the layers that have data determined to be usable as input, without performing calculations on the layers that have data determined to be usable as input. Specifically, the control unit 903 controls the first inference model 1000 to perform calculations on the first layer 1001 and the second layer 1002 that have data determined to be usable as input, without performing calculations on the third layer 1003 that have data determined to be unusable as input. In addition, the control unit 903 controls the second inference model 1020 to perform calculations on the fourth layer 1021 and the fifth layer 1022 that have data determined to be usable as input, without performing calculations on the sixth layer 1023 that have data determined to be unusable as input. By controlling in this way, the resources used for data determined to be usable by the determination unit 902 can be increased by the resources of the second inference model 1020. Furthermore, the second inference model 1020 extracts information about human skin regions that are not extracted by the first inference model 1000, and based on this information, it is possible to perform filtering that suppresses noise removal (averaging processing) so as not to degrade the texture of human skin. As a result, it is possible to improve the image quality of image data that has already been subjected to image quality enhancement processing.

[0095] Next, the image quality improvement process proceeds to S1203. In S1203, the control unit 903 controls the first inference model 1000 to input a fixed value corresponding to the third layer 1003 to the combined layer 1010 instead of the output of the third layer 1003 in the first inference model 1000 that does not perform calculations. The control unit 903 also controls the second inference model 1020 to input a fixed value corresponding to the sixth layer 1023 to the combined layer 1010 instead of the output of the sixth layer 1023 in the second inference model 1020 that does not perform calculations. Next, the image quality improvement process proceeds to S1204.

[0096] In the above-described embodiment, the input data includes image data of interest and reference image data for performing noise reduction to remove noise from the image data of interest. This makes it possible to suppress a decrease in the accuracy of the inference result in the image quality improvement process in which the image data of interest and the reference image data are input.

[0097] Furthermore, in the above-described embodiment, the evaluation value is a value relating to the difference between the image data of interest and the reference image data, so that even if poor reference image data that has a large difference from the image data of interest is input, a decrease in the accuracy of the inference result can be suppressed.

[0098] In addition, in this embodiment, there is no need to match the computational resources used for the second inference model 1020 with the computational resources of the first layer 1001, as long as the amount of computational resources is sufficient to suppress a decrease in the accuracy of the inference results.

[0099] Furthermore, in this embodiment, when determining whether or not input data can be used, determining unit 902 may determine that, among the three input data, only the input data having the largest difference between the evaluation value and the threshold value cannot be used.

[0100] Although the embodiments have been described above in detail, the present invention is not limited to the specific embodiments, and various modifications and changes are possible within the scope of the claims. In addition, it is also possible to combine all or a plurality of the components of the above-described embodiments.

[0101] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0102] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) A control device that performs inference processing using a plurality of related data as input, comprising a first inference model that is composed of a plurality of layers that each receive the plurality of data as input and extract features of the input data, and a combined layer that outputs output data that is an inference result based on the extracted features; a second inference model that receives any of the plurality of data as input and extracts features of the input data; control means for controlling the operation of the first inference model and the second inference model; and judgment means for judging whether each of the plurality of data can be used in the inference processing, wherein the control means judges that all of the plurality of data can be used in the inference processing by the judgment means. a control device which controls the first inference model to output the output data from the coupled layer based on the features extracted in the multiple layers when the determination means determines that the second inference model does not extract features, and controls the first inference model to output the output data from the coupled layer based on the multiple features extracted in the multiple layers, and when the determination means determines that any of the multiple data is unusable for the inference process, the control device controls the first inference model to output the output data from the coupled layer based on the features extracted in a layer of the multiple layers that receives as input data determined by the determination means to be usable for the inference process, and the features extracted by the second inference model that receives as input data determined by the determination means to be usable for the inference process. (Configuration 2) A control device as described in configuration 1, characterized in that when the judgment means judges that all of the multiple data are usable for the inference process, the control means controls so that a predetermined fixed value is input to the coupling layer instead of the output of the second inference model. (Configuration 3) A control device as described in configuration 1 or 2, characterized in that when the judgment means judges that any of the plurality of data is unusable for the inference process, the control means controls so that a predetermined fixed value is input to the coupled layer instead of the output of a layer that inputs data judged by the judgment means to be unusable for the inference process. (Configuration 4) The control device described in any one of configurations 1 to 3, characterized in that the judgment means calculates an evaluation value for each of the multiple data, and judges whether or not the data can be used for the inference process based on the result of comparing the evaluation value with a predetermined threshold value. (Configuration 5) The control device according to any one of configurations 1 to 4, wherein the plurality of data are image data and sound data. (Configuration 6) The control device according to configuration 5, wherein the evaluation value is a value related to any one of the luminance of the image data, the amount of blur of the image data, and the amount of high sensitivity noise of the image data. (Configuration 7) The control device according to configuration 5 or 6, wherein the evaluation value is a value relating to the amount of noise for a feature component of the sound data. (Configuration 8) The control device according to any one of configurations 1 to 4, wherein the plurality of data are a plurality of image data obtained by photographing a person from different angles. (Configuration 9) The control device according to configuration 8, wherein the evaluation value is a value relating to a direction of a face of a person appearing in the image data. (Configuration 10) A control device described in any one of configurations 1 to 4, characterized in that the multiple data include image data of interest and reference image data for performing noise reduction to remove noise from the image data of interest. (Configuration 11) The control device according to configuration 10, wherein the evaluation value is a value related to a difference between the target image data and the reference image data. (Configuration 12) A control device as described in any one of claims 1 to 4, characterized in that the multiple data are image data obtained by photographing a subject, and the determination means determines whether or not the image data can be used for the inference processing based on the shooting conditions at the time of photographing the image data. (Configuration 13) A control device described in any one of configurations 1 to 4, characterized in that the multiple data are two data, and the first inference model has two layers each having the two data as input. (Configuration 14) A control device described in any one of configurations 1 to 4, characterized in that the multiple data are three or more data, and the first inference model has three or more layers, each of which has the multiple data as input. (Configuration 15) The control device described in Configuration 14, characterized in that the judgment means calculates an evaluation value for each of the multiple data and sets a weighting value for each of the multiple data based on the evaluation value, and the control means determines the computational resources of the second inference model based on the weighting value when the judgment means judges that any of the multiple data cannot be used for the inference process. (Configuration 16) A control method for a control device that performs inference processing using multiple related data as input, comprising a control step of controlling the operation of a first inference model that is composed of multiple layers that each receive the multiple data as input and extract features of the input data, and a combined layer that outputs output data that is an inference result based on the extracted features, and a control step of controlling the operation of a second inference model that receives any of the multiple data as input and extracts features of the input data, and a judgment step of judging whether or not each of the multiple data can be used for the inference processing, wherein the control step includes, when it is judged in the judgment step that all of the multiple data can be used for the inference processing, A control method for a control device, characterized in that the second inference model does not extract features, and the first inference model controls to output the output data from the coupled layer based on multiple features extracted in the multiple layers, and when any of the multiple data is determined to be unusable for the inference process in the judgment process, the first inference model controls to output the output data from the coupled layer based on features extracted in a layer of the multiple layers that receives data determined to be usable for the inference process in the judgment process as input, and features extracted by the second inference model using data determined to be usable for the inference process in the judgment process as input. (Configuration 17) A program for causing a computer to execute a control method for a control device that performs inference processing using a plurality of related data as input, the control method for the control device comprising a control step of controlling the operation of a first inference model that is composed of a plurality of layers that each receive the plurality of data as input and extract features of the input data, and a combination layer that outputs output data that is an inference result based on the extracted features, and a control step of controlling the operation of a second inference model that receives any of the plurality of data as input and extracts features of the input data, and a judgment step of judging whether each of the plurality of data can be used for the inference processing, the control step being performed when all of the plurality of data are judged to be usable for the inference processing in the judgment step. and when it is determined in the judgment process that any of the plurality of data is unusable for the inference process, the first inference model controls to output the output data from the combined layer based on features extracted in a layer among the plurality of layers that receives data determined in the judgment process to be usable for the inference process as input, and features extracted by the second inference model that receives data determined in the judgment process to be usable for the inference process as input. (Configuration 18) A method for training a neural network used in a control device described in any one of configurations 1 to 15, comprising the steps of: performing a first learning using a plurality of related learning data, the plurality of learning data being determined by the judgment means to be usable for the inference process, and the first inference model that operates using the plurality of learning data as input; and performing a second learning using a second inference model that operates using, as input, the learning data among the plurality of related learning data determined by the judgment means to be usable for the inference process and the learning data determined to be usable for the inference process, and performing calculations on a layer that receives as input learning data among the plurality of layers determined by the judgment means to be usable for the inference process, and inputting the output of the layer and the output of the second inference model into the coupling layer. (Configuration 19) A neural network training method as described in Configuration 18, characterized in that in the first learning, learning is performed by inputting a predetermined fixed value into the coupling layer instead of the output of the second inference model. (Configuration 20) The neural network learning method described in Configuration 18, characterized in that in the second learning, learning is performed by inputting a predetermined fixed value to the coupling layer instead of the output of a layer that inputs learning data among the plurality of layers that is determined by the determination means to be unusable for the inference process. [Explanation of symbols]

[0103] 100 Control device 102 Judgment section 103 Control section 200 First Inference Model 201 First Layer 202 Second Layer 210 Connecting layer 220 Second Inference Model 900 Control device 902 Judgment section 903 Control Unit 1000 First Inference Model 1001 1st layer 1002 Second Layer 1003 Third Layer 1010 bonding layer 1020 Second Inference Model

Claims

1. A control device that performs inference processing using multiple related data as input, A first inference model comprising multiple layers, each receiving the aforementioned multiple data as input, which extract features from the input data, and a connected layer that outputs output data that constitutes the inference result based on the extracted features, A second inference model takes one of the aforementioned multiple data as input and extracts features from the input data, Control means for controlling the operation of the first inference model and the second inference model, The system includes determination means for determining whether each of the aforementioned plurality of data can be used for the inference process, The control means is If the determination means determines that all of the aforementioned data can be used for the inference process, the second inference model does not perform feature extraction, and the first inference model is controlled to output the output data from the combined layer based on the multiple features extracted by the multiple layers. A control device characterized in that, if any of the multiple data is determined by the determination means to be unusable for the inference process, the first inference model is controlled to output the output data from the combined layer based on the feature quantities extracted by the layer that takes the data determined by the determination means to be usable for the inference process as input, and the feature quantities extracted by the second inference model that takes the data determined by the determination means to be usable for the inference process as input.

2. The control device according to claim 1, characterized in that, if all of the aforementioned plurality of data are determined by the determination means to be usable for the inference process, the control means controls the input of a predetermined fixed value to the coupling layer instead of the output of the second inference model.

3. The control device according to claim 1, characterized in that, if any of the plurality of data is determined by the determination means to be unusable for the inference process, the control means controls the coupling layer to input a predetermined fixed value instead of the output of the layer that takes the data determined by the determination means to be unusable for the inference process as input.

4. The control device according to claim 1, characterized in that the determination means calculates an evaluation value for each of the plurality of data and determines whether or not it can be used for the inference process based on the result of comparing the evaluation value with a predetermined threshold.

5. The control device according to claim 4, characterized in that the plurality of data are image data and sound data.

6. The control device according to claim 5, characterized in that the evaluation value is a value relating to any of the brightness of the image data, the amount of blur of the image data, or the amount of high-sensitivity noise of the image data.

7. The control device according to claim 5, characterized in that the evaluation value is a value relating to the amount of noise with respect to the characteristic components of the sound data.

8. The control device according to claim 4, characterized in that the aforementioned multiple data are multiple image data obtained by photographing a person from different angles.

9. The control device according to claim 8, characterized in that the evaluation value is a value relating to the orientation of the face of a person captured in the image data.

10. The control device according to claim 4, characterized in that the plurality of data include image data of interest and reference image data for performing noise reduction to remove noise from the image data of interest.

11. The control device according to claim 10, characterized in that the evaluation value is a value relating to the difference between the image data of interest and the reference image data.

12. The aforementioned data consists of image data obtained by photographing the subject. The control device according to claim 1, characterized in that the determination means determines whether or not the image data can be used for the inference process based on the shooting conditions at the time of shooting.

13. The aforementioned multiple data sets consist of two data sets, The control device according to claim 1, characterized in that the first inference model comprises two layers, each taking the two data as input.

14. The aforementioned multiple data sets consist of three or more data sets. The control device according to claim 1, characterized in that the first inference model comprises three or more layers, each receiving the plurality of data as input.

15. The determination means calculates an evaluation value for each of the multiple data points, and sets weighting values ​​for each of the multiple data points based on the evaluation values. The control device according to claim 14, wherein the control means determines the computational resources of the second inference model based on the weighting value when any of the plurality of data is determined by the determination means to be unusable for the inference process.

16. A control method for a control device that performs inference processing using multiple related data as input, A control step controls the operation of a first inference model, which is composed of multiple layers that take the aforementioned multiple data as inputs and extract features from the input data, and a connected layer that outputs output data that is an inference result based on the extracted features, and controls the operation of a second inference model that takes any of the aforementioned multiple data as input and extracts features from the input data. The system includes a determination step of determining whether each of the aforementioned plurality of data can be used for the inference process, The control process described above is: If all of the aforementioned data are determined to be usable for the inference process in the determination step, the second inference model does not perform feature extraction, and the first inference model is controlled to output the output data from the combined layer based on the multiple features extracted by the multiple layers. A control method for a control device, characterized in that, if any of the multiple data is determined in the determination step to be unusable for the inference process, the first inference model is controlled to output the output data from the combined layer based on the feature quantities extracted by the layer among the multiple layers that takes the data determined to be usable for the inference process in the determination step as input, and the feature quantities extracted by the second inference model using the data determined to be usable for the inference process in the determination step as input.

17. A program that causes a computer to execute a control method for a control device that performs inference processing using multiple related data as input, The control method for the control device is: A control step controls the operation of a first inference model, which is composed of multiple layers that take the aforementioned multiple data as inputs and extract features from the input data, and a connected layer that outputs output data that is an inference result based on the extracted features, and controls the operation of a second inference model that takes any of the aforementioned multiple data as input and extracts features from the input data. The system includes a determination step of determining whether each of the aforementioned plurality of data can be used for the inference process, The control process described above is: If all of the aforementioned data are determined to be usable for the inference process in the determination step, the second inference model does not perform feature extraction, and the first inference model is controlled to output the output data from the combined layer based on the multiple features extracted by the multiple layers. A program characterized in that, if any of the aforementioned multiple data is determined in the determination step to be unusable for the inference process, the first inference model is controlled to output the output data from the combined layer based on the feature quantities extracted by the layer that takes the data determined to be usable for the inference process in the determination step as input, and the feature quantities extracted by the second inference model using the data determined to be usable for the inference process in the determination step as input.

18. A method for learning a neural network used in a control device according to any one of claims 1 to 15, Multiple related training data sets which are determined by the determination means to be usable for the inference process, A first learning process is performed using the first inference model which operates with the aforementioned plurality of training data as input, Among the multiple related training data, the training data that the determination means determines can be used for the inference process, The second inference model operates using training data determined to be usable for the aforementioned inference process as input, A method for learning a neural network, characterized by performing calculations on a layer that takes learning data determined by the determination means to be usable for the inference process as input from among the plurality of layers, and performing a second learning using the first inference model which operates by inputting the output of that layer and the output of the second inference model to the connected layer.

19. The neural network learning method according to claim 18, characterized in that, in the first learning, learning is performed by inputting predetermined fixed values ​​to the connected layer instead of the output of the second inference model.

20. The neural network learning method according to claim 18, characterized in that, in the second learning, learning is performed by inputting a predetermined fixed value to the connected layer instead of the output of the layer that takes learning data that is determined by the determination means to be unusable for the inference process among the plurality of layers.

Images