Inference apparatus, control method, and program

Abstract

To achieve techniques for improving inference accuracy even when data that is not appropriate for training is included in multiple types of input data.SOLUTION: An inference apparatus includes: inference means that executes inference processing using a plurality of pieces of inference data; determination means that determines whether each of multiple pieces of input data input into the inference apparatus is appropriate for the inference processing; and determination means that determines, in place of input data determined by the determination means to be inappropriate, substitute data based on input data determined by the determination means to be appropriate. The inference means applies the input data determined to be appropriate and the substitute data to an inference model as the multiple pieces of inference data to execute the inference processing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for inputting a plurality of data and executing inference processing. [Background technology]

[0002] Recently, AI (artificial intelligence) has been used for image processing to recognize people's faces and expressions from photographed images. In image processing using AI, photographed images, etc. are used as input data and applied to a trained inference model, inference processing is performed using machine learning, and the inference results are output.

[0003] In this case, if the data used for the inference process is a single input data (single-modal data), it is not sufficient to improve the inference accuracy. Therefore, Patent Document 1 describes a technique for improving the inference accuracy by performing a learning process of an inference model using multiple input data, as compared to performing the learning process using a single input data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-2023 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, if the multiple input data include inappropriate data that is not suitable for learning processing, it becomes difficult to improve the inference accuracy in inference processing using an inference model trained with such inappropriate data.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that improves inference accuracy even when multiple input data include data that is not suitable for learning. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the inference device of the present invention has an inference means for performing an inference process using multiple inference data, a judgment means for judging the suitability of multiple input data for use in the inference process, and a decision means for determining alternative data based on input data judged to be appropriate by the judgment means in place of input data judged to be inappropriate by the judgment means, and the inference means applies the input data judged to be appropriate and the alternative data to an inference model as the multiple inference data to perform the inference process. Effect of the Invention

[0008] According to the present invention, it is possible to improve inference accuracy even when a plurality of input data includes data that is not suitable for learning. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of an inference device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an inference unit according to the first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of image data input to the inference device of the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of sound data input to the inference device of the first embodiment. [Diagram 5] FIG. 4 is a diagram illustrating an example of an aptitude determination result according to the first embodiment. [Figure 6] 5A to 5C are diagrams for explaining a method of calculating characteristic information of input data and a method of determining substitute data according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating inference data according to the first embodiment. [Figure 8] 4 is a flowchart showing a control process according to the first embodiment. [Figure 9] A diagram explaining the learning process of the inference model of embodiment 1. [Figure 10] 13A to 13C are diagrams for explaining a method for calculating characteristic information of input data and a method for determining substitute data according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating inference data according to the second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of image data input to the inference device of the third embodiment. [Figure 13] FIG. 13 is a diagram illustrating inference data according to the third embodiment. [Figure 14] FIG. 13 is a diagram illustrating an example of image data input to the inference device of the fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating inference data according to the fourth embodiment. [Figure 16] FIG. 13 is a block diagram illustrating the configuration of an inference device according to a fifth embodiment. [Figure 17] FIG. 13 is a diagram illustrating the configuration of an inference unit according to the fifth embodiment. [Figure 18] FIG. 13 is a diagram illustrating inference data according to the fifth embodiment. [Figure 19] 13 is a flowchart showing a control process according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [Embodiment 1] A first embodiment will be described with reference to FIGS.

[0012] <Device configuration> First, the configuration and functions of an inference device 100 according to the first embodiment will be described with reference to FIG.

[0013] The inference device 100 has a control unit 101 and an inference unit 102, and executes inference processing using a trained inference model and parameters. The inference device 100 applies multiple input data to the inference model to perform multimodal inference processing, and outputs an inference result.

[0014] In the first embodiment, the inference device 100 receives image data as input 1 and audio data as input 2, and outputs an area focusing on people having a conversation in a captured image as an inference result.

[0015] In the following description, input 1 is image data, input 2 is audio data, and output is a region of interest.

[0016] The inference unit 102 applies the inference data 1 and the inference data 2 output by the control unit 101 to an inference model, performs inference processing by deep learning, and outputs an inference result. The inference result may be output to an internal storage of the inference device 100, or may be output to an external device via a wide area network such as the Internet.

[0017] The suitability determination unit 103 determines the suitability of image data and sound data as input data when the inference unit 102 performs inference processing. The suitability determination unit 103 calculates evaluation information of the image data and sound data, and compares the evaluation information with a threshold value. The suitability determination unit 103 determines input data whose evaluation information is equal to or greater than the threshold value as suitable for inference processing (hereinafter, "suitable"), and determines input data whose evaluation information is less than the threshold value as not suitable for inference processing (hereinafter, "inappropriate"), and outputs the determination result to the control unit 101. The method of determining the suitability of input data will be described in detail later.

[0018] The characteristic information calculation unit 104 acquires the characteristics of the image data and the sound data, and calculates normalized characteristic values ​​according to the characteristics. The method of calculating the characteristic values ​​will be described in detail later.

[0019] The control unit 101 has a processor such as a CPU, and memories such as a ROM and a RAM. The control unit 101 controls data for inference to be output to the inference unit 102 based on the characteristic values ​​of image data and sound data. The control unit 101 determines substitute data for input data determined to be inappropriate by the suitability determination unit 103 based on the characteristic values ​​of image data and sound data. When the suitability determination unit 103 determines that the image data and sound data are appropriate, the control unit 101 outputs the image data as data for inference 1 and the sound data as data for inference 2 to the inference unit 102. When the suitability determination unit 103 determines that the image data is appropriate and the sound data is inappropriate, the control unit 101 outputs the image data as data for inference 1 and the substitute data for the sound data as data for inference 2 to the inference unit 102. Details of the method of controlling the data for inference will be described later.

[0020] <Configuration of inference unit 102> Next, the configuration and functions of the inference unit 102 of this embodiment will be described with reference to FIG.

[0021] The inference unit 102 has an inference model 200. The inference model 200 is configured by a neural network, and in this embodiment, is configured by a convolutional neural network (CNN). Note that the inference model in this embodiment is not limited to CNN, and may be configured by a neural network such as an RNN (Recurrent Neural Network) or a fully connected type. The calculation method of the neural network is well known, so a description thereof will be omitted.

[0022] The inference process by deep learning in the inference unit 102 can be executed by a GPU (Graphics Processing Unit). The same applies to the learning process of the inference model 200 described later in FIG. 9. The GPU is a processor capable of performing processes specialized for computer graphic calculations, and has the calculation processing capacity to perform matrix calculations and the like required for deep learning in a short time. Note that the deep learning may be performed by the CPU of the control unit 101 and the GPU of the inference unit 102 working together, or may be performed by either the CPU of the control unit 101 or the GPU of the inference unit 102.

[0023] The inference model 200 has a first layer 211 that receives inference data 1 output by the control unit 101 as input, a second layer 212 that receives inference data 2 output by the control unit 101 as input, and a combination layer 210 that combines the output of the first layer 211 and the output of the second layer 212 to output the result of inferring the area of ​​interest.

[0024] The first layer 211 extracts features related to the inference data 1. The first layer 211 extracts the likelihood of a person being a person of interest obtained from an image, for example.

[0025] The second layer 212 extracts features related to the inference data 2. The second layer 212 extracts the likelihood of a person being a person of interest obtained from, for example, a voice.

[0026] The combination layer 210 receives the output of the first layer 211 and the output of the second layer 212, combines the features output from the first layer 211 and the features output from the second layer 212, and outputs an inference result.

[0027] <Explanation of image data> Next, image data input to the inference device 100 of this embodiment will be described with reference to FIG.

[0028] FIG. 3(a) shows an example of Image 0 that includes a person talking and a person walking, has no blur, and has some image noise.

[0029] FIG. 3B illustrates an example of an image 1 that is more blurred and noisy than the image 0.

[0030] In the present embodiment, the image noise is high-sensitivity noise that occurs due to a high ISO sensitivity when capturing an image, but is not limited to this and may be long-term noise that occurs when the image sensor heats up, or the like.

[0031] In the present embodiment, the blur of an image refers to blur caused by camera shake or subject motion when capturing an image, or blur caused by focus shift, but is not limited to these and may also be blur caused by the optical system when capturing an image.

[0032] The method of calculating the amount of noise and the amount of blur is well known, and therefore the explanation thereof will be omitted.

[0033] <Explanation of sound data> Next, with reference to FIG. 4, the sound data input to the inference device 100 of this embodiment will be described.

[0034] FIG. 4(a) illustrates an example of the waveform of sound 0, which is the voice of a person speaking, among the sound data input to the inference device 100.

[0035] FIG. 4(b) illustrates an example of a noise waveform that is generated around the inference device 100 and includes no noise such as wind noise.

[0036] FIG. 4(c) illustrates an example of a sound waveform of sound 1, in which the volume of speech of a person speaking is small and the amplitude of the sound is smaller than sound 0, in comparison with sound 0 in FIG. 4(a).

[0037] Figure 4(d) illustrates an example of a noise sound, Sound 1, and illustrates the waveform of the noise sound Sound 1, which is louder than Sound 0 due to environmental sounds occurring around the inference device 100, such as the sound of a car driving.

[0038] The method of calculating the volume and the amount of noise is well known, and therefore the explanation is omitted.

[0039] <Results of suitability assessment of input data> Next, a method for determining the suitability of image data and sound data by the suitability determining unit 103 of this embodiment will be described with reference to FIG.

[0040] The suitability determination unit 103 calculates evaluation information for determining the suitability of image data and sound data as input data within a range of evaluation values ​​from a minimum of 0 to a maximum of 4. In this embodiment, the threshold value is set to 2 to determine the suitability of the input data. The suitability determination unit 103 determines that an evaluation value of 2 or more is appropriate, and that an evaluation value of less than 2 is unusable.

[0041] In this embodiment, the larger the evaluation value, the more suitable the data is for inference processing, and the smaller the evaluation value, the less suitable the data is for inference processing, and suitable data and inappropriate data are determined based on threshold value 2. Note that the method for determining the suitability of input data is not limited to the above-mentioned method, and any method may be used.

[0042] The suitability determination unit 103 determines the suitability of input image data based on the amount of blur, and determines the suitability of input sound data based on the volume.

[0043] Here, a method for calculating an evaluation value relating to the amount of blur of image data and an evaluation value relating to the volume of sound data will be described.

[0044] First, a method for calculating an evaluation value of the amount of blur of image data will be described.

[0045] In this embodiment, the object in the image is recognized to identify the person who is speaking and obtain the attention area. Therefore, the greater the amount of blur in the image data, the more difficult it becomes to identify the person who is speaking, so the amount of blur in the image data is used as the evaluation value to determine suitability. For example, if the amount of blur in image data that is so blurred that a person cannot be identified is set to 100%, and the amount of blur in image data with no blur is set to 0%, the greater the amount of blur, the lower the evaluation value is set, and the smaller the amount of blur, the higher the evaluation value is set. In this embodiment, the evaluation value is set to 0 to 4 for every 20% of the amount of blur.

[0046] The threshold value of the amount of blur in this embodiment is a threshold value for determining whether or not a person can be identified.

[0047] Next, a method for calculating an evaluation value of the volume of sound data will be described.

[0048] In this embodiment, when recognizing a subject in an image, since it is difficult to obtain the voice of a person who is speaking if the volume is low, the volume of the sound data is used as an evaluation value to determine suitability. For example, if the volume of sound data with no conversation volume is set to 0% and the limit value of the volume of a sound collection device such as a microphone that obtains sound is set to 100%, the evaluation value is set lower as the volume is lower and higher as the volume is higher. In this embodiment, the evaluation value is set to 0 to 4 for every 20% of the volume. The volume threshold in this embodiment is a threshold for determining whether or not the voice of a person who is speaking can be identified. Note that if the volume is too loud to identify the conversation, the evaluation value may be lowered.

[0049] In this embodiment, the evaluation value is calculated based on the amount of blur in the image data and the volume of the audio data, and the suitability is determined based on the evaluation value, but the evaluation value may be based on brightness, high sensitivity noise, or noise such as ambient noise. Note that the method of calculating the evaluation value is well known, so a description thereof will be omitted.

[0050] FIG. 5(a) illustrates the results of determining suitability for each of images 0 and 1 as input data, based on the evaluation value calculated from the amount of blur and a threshold value.

[0051] When image 0 is input, suitability judgment unit 103 determines that image 0 is appropriate because there is no blur and the evaluation value of the amount of blur is 4, which is equal to or greater than threshold value 2. When image 1 is input, suitability judgment unit 103 determines that image 1 is appropriate because there is a lot of blur and the evaluation value is 1, which is less than threshold value 2.

[0052] FIG. 5(b) illustrates the results of determining the suitability of each of sounds 0 and 1 as input data based on the volume and threshold value.

[0053] When sound 0 is input, suitability determination unit 103 determines that the sound is appropriate by setting the evaluation value of the volume to 3, which is equal to or greater than threshold value 2. When sound 1 is input, suitability determination unit 103 determines that the sound is appropriate by setting the evaluation value to 0, which is less than threshold value 2, because the sound waveform cannot be obtained.

[0054] <How to determine characteristic values ​​of input data and alternative data> Next, a method for calculating characteristic information of input data and a method for determining substitute data will be described with reference to FIG.

[0055] 6(a) shows an example of characteristic values ​​of input data and alternative data. The characteristic information calculation unit 104 calculates characteristic values ​​with characteristics obtainable from the input data ranging from a minimum of 0 to a maximum of 9. The control unit 101 determines alternative data 2 when the characteristic value is 6 to 9, alternative data 1 when the characteristic value is 3 to 5, and alternative data 0 when the characteristic value is 0 to 2.

[0056] In this embodiment, the characteristics of the high sensitivity noise amount in the image data are obtained, and a normalized characteristic value is calculated.

[0057] Here, a normalization method according to the characteristics of the high sensitivity noise amount in the image data will be described.

[0058] The more high-sensitivity noise there is in the image data, the more difficult it becomes to identify the person who is talking as the person of interest. For example, if the noise amount of image data with so much high-sensitivity noise that the person cannot be identified is set to 100%, and the noise amount of an image with no noise is set to 0%, the characteristic value is set lower as the amount of high-sensitivity noise increases, and higher as the amount of high-sensitivity noise decreases. In this embodiment, the characteristic value is calculated as 0 to 9 for every 10% of the amount of noise.

[0059] 6(b) illustrates an example of characteristic values ​​of high sensitivity noise in images 0 and 1 when images 0 and 1 in FIG. 3 are used as input data, and substitute data determined based on the characteristic values. Note that, in this embodiment, high sensitivity noise is illustrated as an example of the characteristic value calculated by the characteristic information calculation unit 104, but the characteristic value may be the S / N ratio, brightness, amount of blur, etc. in the image data.

[0060] The information calculated by the suitability determination unit 103 and the characteristic information calculation unit 104 may be the same. In this case, the information is the amount of blurring and high-sensitivity noise in this embodiment. For example, in this embodiment, both the suitability determination unit 103 and the characteristic information calculation unit 104 may calculate the evaluation value and the characteristic value using the amount of blurring.

[0061] When image 0 is input, characteristic information calculation unit 104 calculates the characteristic value of high sensitivity noise to be 7, and control unit 101 determines alternative data to be 2 since the characteristic value is 6 to 9. When image 1 is input, characteristic information calculation unit 104 calculates the characteristic value of high sensitivity noise to be 4, and control unit 101 determines alternative data to be 1 since the characteristic value is 3 to 5.

[0062] Figure 6(c) shows an example of characteristic values ​​of noise sounds, which are environmental sounds around the inference device 100, when sounds 0 and 1 in Figure 3 are used as input data, and alternative data determined by the control unit 101 based on the characteristic values.

[0063] When sound 0 is input, the characteristic information calculation unit 104 calculates 5 as the characteristic value of the noise sound, and the control unit 101 determines the alternative data to be 1 since the characteristic value is between 3 and 5. When sound 1 is input, the characteristic information calculation unit 104 calculates 2 as the characteristic value of the noise sound, and the control unit 101 determines the alternative data to be 0 since the characteristic value is between 0 and 2.

[0064] The normalization method is not limited to the above-mentioned method, and any method that can calculate a characteristic value quantified based on a certain standard may be used. In the present embodiment, the characteristic value is set to 10 levels from 0 to 9, and three types of alternative data are set, but the normalization method is not limited to this.

[0065] Furthermore, the characteristic information calculation unit 104 may calculate the characteristic values ​​only for the data that is determined to be appropriate by the suitability determination unit 103. This makes it possible to reduce the time required for the calculation process of the characteristic information calculation unit 104.

[0066] <How to determine data for inference> Next, with reference to FIG. 7, a method for determining the inference data to be output from the control unit 101 to the inference unit 102 based on the suitability judgment result of the suitability judgment unit 103 and the alternative data determined by the control unit 101 will be described.

[0067] In Figure 7, when image 0 is input to input 1 and sound 0 is input to input 2, both input data are appropriate, so the control unit 101 outputs image 0 as inference data 1 and sound 0 as inference data 2.

[0068] When image 0 is input to input 1 and sound 1 is input to input 2, image 0 is appropriate but sound 1 is inappropriate, so the control unit 101 outputs image 0 as inference data 1 and image alternative data 2 as inference data 2 in place of sound 1.

[0069] When image 1 is input as input 1 and sound 0 is input as input 2, image 1 is inappropriate but sound 0 is appropriate, so the control unit 101 outputs sound alternative data 1 as inference data 1 and sound 0 as inference data 2.

[0070] When image 1 is input to input 1 and sound 1 is input to input 2, both input data are inappropriate, so suitability judgment unit 103 again compares the evaluation values ​​used when judging the suitability of image 1 and sound 1, and appropriately changes the suitability judgment result of image 1, which has the higher evaluation value. Control unit 101 outputs image 1 as inference data 1 and image substitute data 1 as inference data 2.

[0071] <Control processing of inference data> Next, control processing of inference data by the inference device 100 of this embodiment will be described with reference to FIG.

[0072] The processing in FIG. 8 is realized by the control unit 101 executing a program stored in memory, and is started when image data and sound data are input to the inference device 100.

[0073] In step S801, the suitability determining unit 103 calculates an evaluation value of the input data.

[0074] In step S802, the suitability determination unit 103 compares the evaluation value calculated in step S801 with a threshold value. If the suitability determination unit 103 determines that the evaluation value is equal to or greater than the threshold value, it determines that the input data is appropriate, and if it determines that the evaluation value is less than the threshold value, it determines that the input data is inappropriate. If the suitability determination unit 103 determines that all of the input data is appropriate, it proceeds to step S803. If the suitability determination unit 103 determines that some of the input data is inappropriate, it proceeds to step S805, and if it determines that all of the input data is inappropriate, it proceeds to step S807.

[0075] In step S803, the control unit 101 outputs the image data as inference use data 1 and the sound data as inference use data 2.

[0076] In step S804, the inference unit 102 performs inference processing using the inference data 1 and inference data 2 output from the control unit 101, outputs the inference result, and ends the processing.

[0077] In step S805, the control unit 101 determines alternative data for the input data determined to be inappropriate in step S802, based on the characteristic values ​​of the input data determined to be appropriate in step S802.

[0078] In step S806, the control unit 101 outputs the input data determined to be appropriate in step S802 as inference data 1, and the alternative data determined in S805 as inference data 2, and proceeds to step S804.

[0079] In step S807, the suitability judgment unit 103 appropriately changes the suitability judgment result of the input data with the highest evaluation value among the input data judged to be inappropriate, and proceeds to step S805.

[0080] According to embodiment 1, when a portion of the input data is judged to be inappropriate, alternative data determined based on the characteristic values ​​of the data judged to be appropriate is used as the data for inference in place of the input data judged to be inappropriate, thereby making it possible to improve the inference accuracy compared to the inference process using the input data judged to be inappropriate.

[0081] Furthermore, if all of the input data is determined to be appropriate, the input data determined to be appropriate is used as data for inference, making it possible to prevent a decrease in inference accuracy.

[0082] In addition, if all input data is judged to be inappropriate, the suitability of the input data with the highest evaluation value among the input data judged to be inappropriate is appropriately changed, and alternative data determined based on the characteristic value of the input data with the highest evaluation value is used as the inference data in place of the other input data judged to be inappropriate, thereby making it possible to improve the inference accuracy compared to an inference process using input data all judged to be inappropriate.

[0083] <Learning process for inference model> Next, the learning process of the inference model 200 in FIG. 2 will be described with reference to FIG.

[0084] The inference model 200 of the inference model unit 102 in this embodiment has undergone learning processing in advance by an information processing device such as a personal computer (PC) different from the inference device 100.

[0085] The learning process of the inference model 200 can be the same as that of the inference unit 102 of the inference device 100.

[0086] The learning process of the inference model 200 in this embodiment is a process of optimizing the parameters of the inference model 200 using previously prepared learning data, and updating the parameters of the inference model 200 to the optimized parameters. The learning data includes input data and teacher data. The input data is input to the inference model 200, and the parameter adjustment unit 300 optimizes the parameters of the inference model 200 so that the deviation between the output data of the inference model 200 and the teacher data is minimized. The inference model 200 obtains the optimized parameters from the parameter adjustment unit 300 and updates the existing parameters.

[0087] The learning data includes learning input data and teacher data. The learning input data is a data set prepared by combining a plurality of data, and in this embodiment, it is inference data 1 and inference data 2. The first data set is a data set combining image data and sound data determined to be appropriate by the suitability judgment unit 103. The second data set is a data set combining image data determined to be appropriate by the suitability judgment unit 103 and alternative data determined based on the characteristic value of the image data. The third data set is a data set combining sound data determined to be appropriate by the suitability judgment unit 103 and alternative data determined based on the characteristic value of the sound data.

[0088] The teacher data for learning is output data output by a preliminary learning process using input data for learning, and in this embodiment, it is data of a region of interest output by a learning process using image data and sound data as inference data as input data.

[0089] By using the above learning data in the learning process of the inference model 200, the learning process is limited to input data that is a combination of input data and alternative data determined based on the characteristic values ​​of the input data, making it possible to reduce the number of parameters and the amount of data.

[0090] The input data for learning is not limited to image data or sound data, but may be any data that can provide evaluation values ​​and characteristic values ​​of the data.

[0091] In addition, the input data for learning is not limited to two, inference data 1 and inference data 2, but may be three or more input data.In this case, the inference model 200 may have a processing layer corresponding to multiple input data and perform multiple learning processes.

[0092] Note that since the learning algorithm of the neural network, such as the backpropagation method, is well known, a description thereof will be omitted.

[0093] [Embodiment 2] Next, a second embodiment will be described with reference to FIGS.

[0094] The second embodiment is different from the first embodiment in the method of calculating the characteristic value by the characteristic information calculation unit 104 and the method of determining substitute data by the control unit 101.

[0095] The configuration of the inference device 100, and the configurations and functions of the inference unit 102 and the suitability determination unit 103 of the second embodiment are the same as those of the first embodiment.

[0096] The characteristic value calculation unit 104 obtains a plurality of characteristics from the image data and the sound data, and calculates a plurality of normalized characteristic values ​​for each characteristic.

[0097] The control unit 101 determines substitute data associated with a plurality of characteristic values ​​based on the plurality of characteristic values ​​of the image data and sound data, and controls the data for inference to be output to the inference unit 102 based on the suitability judgment result.

[0098] <How to determine characteristic values ​​of input data and alternative data> Next, with reference to FIG. 10, a method in the second embodiment in which the characteristic value calculation unit 104 calculates two characteristic values ​​from one input data and the control unit 101 determines substitute data based on the two characteristic values ​​will be described.

[0099] FIG. 10(a) illustrates a first characteristic value and a second characteristic value calculated from one piece of input data, and substitute data determined by the control unit 101.

[0100] For example, if the first characteristic value is in the range of 0 to 2 and the second characteristic value is in the range of 0 to 2, the alternative data is determined to be 00. If the first characteristic value is in the range of 0 to 2 and the second characteristic value is in the range of 3 to 5, the alternative data is determined to be 01.

[0101] In this manner, one alternative data is determined based on a plurality of characteristic values.

[0102] FIG. 10(b) illustrates an example of alternative data determined by the control unit 101 from the relationship in FIG. 10(a) based on a first characteristic value calculated from the amount of high-sensitivity noise in the image data and a second characteristic value calculated from the amount of blur in the image data.

[0103] When image 0 is input, the characteristic information calculation unit 104 calculates 7 as the characteristic value of the high sensitivity noise amount and 8 as the characteristic value of the blur amount, and the control unit 101 determines the substitute data for image 0 to be substitute data 22.

[0104] When image 1 is input, characteristic information calculation unit 104 calculates a characteristic value of 4 for the amount of high sensitivity noise and a characteristic value of 2 for the amount of blur, and control unit 101 determines substitute data 10 as the substitute data for image 1.

[0105] FIG. 10(c) illustrates an example of alternative data determined by the control unit 101 from the relationship in FIG. 10(a) based on a first characteristic value calculated from the noise sound of the sound data and a second characteristic value calculated from the volume of the sound data.

[0106] When sound 0 is input, the characteristic information calculation unit 104 calculates the characteristic value of the noise sound as 5 and the characteristic value of the volume as 6, and the control unit 101 determines the alternative data for sound 0 to be alternative data 21.

[0107] When sound 1 is input, the characteristic information calculation unit 104 calculates 2 as the characteristic value of the noise sound and 1 as the characteristic value of the volume, and the control unit 101 determines the alternative data for sound 1 to be alternative data 01.

[0108] <How to determine data for inference> Next, with reference to FIG. 11, a method in which the control unit 101 determines data for inference based on the suitability judgment result of the suitability judgment unit 103 and the alternative data determined by the characteristic information calculation unit 104 in embodiment 2 will be described.

[0109] It should be noted that the suitability judgment result of the input data in the second embodiment is the same as that in the first embodiment.

[0110] Figure 11 is a diagram explaining the method by which the control unit 101 determines data for inference based on the suitability judgment result of Figure 5 and the alternative data determined in Figure 10 when image 0 or image 1 is input as input 1 and sound 0 or sound 1 is input as input 2 to the inference device 100.

[0111] When image 0 is input as input 1 and sound 0 is input as input 2, the suitability judgment results are both appropriate, so the control unit 101 outputs image 0 as data for inference 1 and sound 0 as data for inference 2.

[0112] When image 0 is input to input 1 and sound 1 is input to input 2, image 0 is appropriate but sound 1 is inappropriate, so the control unit 101 outputs image 0 as inference data 1 and image alternative data 22 as inference data 2 in place of sound 1.

[0113] When image 1 is input to input 1 and sound 0 is input to input 2, image 1 is inappropriate and sound 0 is appropriate, so the control unit 101 outputs sound 0 as inference data 2 and sound alternative data 21 instead of image 1 as inference data 1.

[0114] When image 1 is input as input 1 and sound 1 is input as input 2, both suitability judgment results are inappropriate, so suitability judgment unit 103 compares the evaluation values ​​when judging the suitability of image 1 and sound 1, and appropriately changes the suitability judgment result of image 1, which has the higher evaluation value. Control unit 101 outputs image 1 as inference data 1 and image substitute data 10 instead of sound 1 as inference data 2. According to the second embodiment, inference processing is performed using input data and substitute data determined based on a plurality of characteristic values, thereby making it possible to further improve inference accuracy compared to the first embodiment.

[0115] <Learning process for inference model> Next, the learning process of the inference model of embodiment 2 will be described.

[0116] The learning process of the inference model 200 in this embodiment differs from the learning data used in the learning process of embodiment 1 in that the input data for learning is different, but the rest is the same as embodiment 1, so a description will be omitted.

[0117] The learning input data is a data set prepared by combining multiple data, and in this embodiment, it is inference data 1 and inference data 2. The first data set is a data set combining image data and sound data that the suitability judgment unit 103 judges to be appropriate. The second data set is a data set combining image data that the suitability judgment unit 103 judges to be appropriate and alternative data determined based on multiple characteristic values ​​of the image data. The third data set is a data set combining sound data that the suitability judgment unit 103 judges to be appropriate and alternative data determined based on multiple characteristic values ​​of the sound data.

[0118] By using the above learning data in the learning process of the inference model 200, the learning process is limited to input data that is a combination of the input data and alternative data determined based on multiple characteristic values ​​of the input data, making it possible to reduce the number of parameters and the amount of data. Note that the more input data for learning is prepared and used for learning, the greater the effect of reducing the amount of data. Data is not limited to image data or sound data, and can be any data from which evaluation values ​​and characteristic values ​​can be obtained.

[0119] In addition, the input data for learning is not limited to two, inference data 1 and inference data 2, but may be three or more input data.In this case, the inference model 200 may have a processing layer corresponding to multiple input data and perform multiple learning processes.

[0120] Note that since the learning algorithm of the neural network, such as the backpropagation method, is well known, a description thereof will be omitted.

[0121] [Embodiment 3] Next, a third embodiment will be described with reference to FIGS.

[0122] The third embodiment is different from the first embodiment in the image data used as input data.

[0123] The configuration of the inference device 100, and the configurations and functions of the inference unit 102 and characteristic information calculation unit 104 of the third embodiment are the same as those of the first embodiment.

[0124] The inference device 100 of the third embodiment identifies a person by performing inference processing using images of a subject photographed from different directions or angles as input data. Both data for inference 1 and data for inference 2 are input as image data, and output data is specific person information. Note that the specific person information to be output may be image or text information, and may be any information that can identify a person.

[0125] <Description of input data> Next, image data input to the inference device 100 of the third embodiment will be described with reference to FIG.

[0126] FIG. 12 illustrates images of a person's face taken from different directions or angles.

[0127] FIG. 12(a) illustrates an example of a front image taken from the front.

[0128] FIG. 12(b) shows an example of a landscape image taken from directly beside the camera, which has more high-sensitivity noise than the frontal image in FIG. 12(a).

[0129] FIG. 12(c) shows an example of a rear-facing image taken from directly behind, which has more high-sensitivity noise than the side-facing image in FIG. 12(b).

[0130] In the third embodiment, the suitability determination unit 103 calculates an evaluation value according to the amount of facial feature information, such as the eyes, nose, and mouth, of the face of a person in an image.

[0131] For example, a state in which the face is facing straight ahead and the eyes, nose, mouth, etc. are all included in the image is rated as 100%, and a state in which the face is facing directly behind the subject and the eyes, nose, mouth, etc. are not included in the image is rated as 0%, and an evaluation value is calculated in the range of 0 to 4 according to the amount of feature information.Then, the evaluation value is compared with a threshold value to determine the suitability of the data.

[0132] Here, a method in which the suitability determining unit 103 calculates an evaluation value of image data when the image in FIG. 12 is input will be described.

[0133] FIG. 12(a) is a frontal image in which the eyes, nose, mouth, etc. are all included in the image, so an evaluation value of 4 is calculated.

[0134] FIG. 12(b) is a sideways image, and the amount of feature information of the eyes, nose, mouth, etc. is about 50% of that of the frontal image, so an evaluation value of 2 is calculated.

[0135] FIG. 12(c) is a rear-facing image, and since the eyes, nose, mouth, etc. are not included in the image, an evaluation value of 0 is calculated.

[0136] <How to determine data for inference> Next, with reference to FIG. 13, a method in which the control unit 101 determines data for inference based on the suitability judgment result of the suitability judgment unit 103 and the alternative data determined by the characteristic information calculation unit 104 in embodiment 3 will be described.

[0137] FIG. 13(a) illustrates the evaluation value and suitability judgment result for each input data.

[0138] When a frontal image is input, the evaluation value is 4, which is greater than or equal to the threshold value 2, and is therefore determined to be appropriate. When a sideways image is input, the evaluation value is 3, which is greater than or equal to the threshold value 2, and is therefore determined to be appropriate. When a rearward-facing image is input, the evaluation value is 0, which is less than the threshold value 2, and is therefore determined to be inappropriate.

[0139] 13B illustrates an example of substitute data that is determined based on a characteristic value calculated for each input data. In this embodiment, high sensitivity noise is calculated as the characteristic value, and substitute data is determined based on the characteristic value.

[0140] When a frontal image is input, the characteristic information calculation unit 104 calculates a characteristic value of 9, and the control unit 101 determines the alternative data to be 2. When a sideways image is input, the characteristic information calculation unit 104 calculates a characteristic value of 6, and the control unit 101 determines the alternative data to be 1. When a rearward facing image is input, the characteristic information calculation unit 104 calculates a characteristic value of 0, and the control unit 101 determines the alternative data to be 0.

[0141] Figure 13(c) is a diagram explaining how the control unit 101 determines inference data 1 and inference data 2 based on the suitability judgment result in Figure 13(a) and the alternative data determined in Figure 13(b) when a frontal image or a sideways image is input as input 1 and a sideways image or a rearward image is input as input 2 to the inference device 100.

[0142] When a front image is input as input 1 and a landscape image is input as input 2, both are appropriate, so the control unit 101 outputs the front image as inference data 1 and the landscape image as inference data 2.

[0143] When a landscape image is input as input 1 and a rear-facing image is input as input 2, the rear-facing image is determined to be inappropriate, so the control unit 101 outputs the landscape image as inference data 1 and image alternative data 1 as inference data 2.

[0144] According to embodiment 3, when the amount of characteristic information of some input data is small, an evaluation value is calculated based on the amount of characteristic information to judge the suitability of the data, and alternative data determined based on the characteristic values ​​of data judged to be appropriate is used in the inference process instead of the data judged to be inappropriate, thereby making it possible to improve the inference accuracy compared to the inference process using data judged to be inappropriate.

[0145] [Embodiment 4] Next, a fourth embodiment will be described with reference to FIGS.

[0146] The fourth embodiment is different from the first embodiment in the image data used as input data.

[0147] The configuration of the inference device 100, and the configurations and functions of the inference unit 102 and characteristic information calculation unit 104 of the fourth embodiment are the same as those of the first embodiment.

[0148] The inference device 100 of the fourth embodiment performs image quality improvement processing on a subsequent image of interest in a plurality of image data captured continuously, using a previous image as a reference image. The latest image data is input as the image of interest to input 1, image data of the immediately previous frame is input as the reference image to input 2, and the output data is image data after the image quality of the image of interest has been improved.

[0149] The improvement in image quality is a noise reduction process, and for example, by performing the noise reduction process using multiple pieces of data, it is possible to remove noise with higher accuracy. Note that there are multiple methods for the improvement in image quality process, but since all of them are well known, the description thereof will be omitted.

[0150] <Description of input data> Next, image data input to the inference device 100 of the fourth embodiment will be described with reference to FIG.

[0151] FIG. 14 illustrates an example of continuously captured images in chronological order.

[0152] FIG. 14(a) illustrates an example of an image a in which a person is located at the center of the image.

[0153] FIG. 14(b) shows an example of image b, which is the frame next to image a in FIG. 14(a). The position of the person's face is the same as in image a, but there is some high-sensitivity noise.

[0154] FIG. 14(c) shows an example of an image c, which is the frame following image b in FIG. 14(b), in which the position of the person's face is the same as in image a, but an example of an image in which a flash is fired.

[0155] The suitability determination unit 103 calculates the difference between the image of interest and the reference image for the continuously captured image data, and determines suitability based on the result of comparing the difference with a threshold. In the fourth embodiment, the difference between the image of interest and the reference image is the luminance difference.

[0156] The difference between the image of interest and the reference image may be calculated based on the average or variance of a predetermined region. In this case, the difference between the images may be calculated after predicting the movement between the images and aligning them.

[0157] <How to determine data for inference> Next, referring to Figure 15, we will explain how the control unit 101 determines the inference data to be output to the inference unit 102 based on the suitability judgment result of the suitability judgment unit 103 and the alternative data determined by the characteristic information calculation unit 104.

[0158] FIG. 15A illustrates an example of characteristic values ​​of high-sensitivity noise calculated by the characteristic information calculation unit 104 from images a, b, and c, and alternative data determined by the control unit 101 from the characteristic values.

[0159] Since image a does not have high-sensitivity noise, characteristic information calculation section 104 sets the characteristic value to 9, and control section 101 determines alternative data to be 2 based on the characteristic value.

[0160] Since images b and c contain some high-sensitivity noise, characteristic information calculation section 104 sets the characteristic value to 6, and control section 101 determines substitute data 1.

[0161] The characteristic value of the high sensitivity noise is calculated in the same manner as in the first embodiment, and the greater the amount of high sensitivity noise, the lower the characteristic value is calculated, and the smaller the amount of high sensitivity noise, the higher the characteristic value is calculated.

[0162] 15(b) illustrates an example of substitute data determined by the suitability determination unit 103 based on the evaluation value of the luminance difference between the images and the control unit 101 based on the suitability determination result and characteristic values ​​when the target image input 1 and the reference image input 2 are input in the order of image a, image b, and image c in chronological order. In this embodiment, the suitability is determined using a threshold value of 2.

[0163] When image a is input, since there is no reference image, suitability judgment unit 103 does not perform suitability judgment, and control unit 101 outputs image a as inference data 1 and image substitute data 2 as inference data 2.

[0164] When image b is input to input 1 and image a is input to input 2, suitability determination unit 103 sets the evaluation value of image b to 4 because there is no difference in luminance between images b and a, and determines image b to be appropriate because it is equal to or greater than threshold value 2. Control unit 101 outputs image b as inference data 1 and image a as inference data 2. In this case, the difference in luminance between the image of interest and the reference image is small, making it possible to perform intended noise reduction and improve the image quality.

[0165] When image c is input to input 1 and image b is input to input 2, image c has a high brightness due to the flash and has a large brightness difference with image b, so that when high image quality processing is performed on the input image data, the inference accuracy decreases and the intended noise reduction cannot be performed. Therefore, the suitability determination unit 103 sets the evaluation value of image c to 0 because the brightness difference between images c and b is large, and determines image c to be inappropriate because it is less than threshold value 2. The control unit 101 outputs image substitute data 1 as inference data 1 and image b as inference data 2.

[0166] According to the fourth embodiment, data in which the brightness of some parts of continuously captured image data is high and the brightness difference between the previous and following image data is large is judged to be inappropriate, and alternative data determined based on data judged to be appropriate is used in the inference process instead of the data judged to be inappropriate, thereby making it possible to improve the inference accuracy compared to the inference process using data judged to be inappropriate.

[0167] [Embodiment 5] Next, a fifth embodiment will be described with reference to FIGS.

[0168] In the fifth embodiment, the differences from the first to fourth embodiments will be mainly described, and descriptions of the same or similar parts to the first to fourth embodiments will be omitted.

[0169] <Device configuration> First, the configuration and functions of an inference device 1600 according to the fifth embodiment will be described with reference to FIG.

[0170] The differences from the first embodiment are the addition of input 3 and the processing of each part.

[0171] The inference device 1600 of the fifth embodiment takes image data as input 1, image data as input 2, and audio data as input 3, and outputs an area focusing on people having a conversation in the image as an inference result. Note that the number of types of input data is not limited to three, and may be three or more.

[0172] Examples of input data in the fifth embodiment are image 0, image 1, sound 0, and sound 2 described in Fig. 3 and Fig. 4. Note that sound 2 is data of the same sound source as sound 1 in the first embodiment, but has a larger amplitude than sound 1.

[0173] The inference unit 1602 applies the inference data 1, inference data 2, and inference data 3 output by the control unit 1601 to an inference model, performs inference processing using deep learning, and outputs the inference result.

[0174] The suitability determination unit 1603 calculates two evaluation values ​​for determining the suitability of image data and sound data for each of input 1, input 2, and input 3, and determines the suitability by comparing the evaluation values ​​with two thresholds. The method of determining suitability by comparison with the first threshold is the same as in the first embodiment, and input data whose evaluation value is equal to or greater than the first threshold is determined to be suitable, and input data whose evaluation value is less than the first threshold is determined to be inappropriate.

[0175] In the method of determining suitability by comparison with the second threshold, if the difference between the input data with the highest evaluation value and the input data with the second highest evaluation value is equal to or greater than the second threshold, the input data with the highest evaluation value is determined to be appropriate, and the other input data is determined to be inappropriate. If the difference in the evaluation values ​​is less than the second threshold, all input data is determined to be appropriate, and the suitability determination result is output to the control unit 1601. Details of the suitability determination method will be described later.

[0176] The characteristic calculation unit 1604 acquires the characteristics of each of the data of the input 1, the input 2, and the input 3, and calculates a plurality of normalized characteristic values ​​for each characteristic.

[0177] The control unit 1601 controls the data for inference to be output to the inference unit 1602 based on the suitability judgment results and characteristic values ​​of each data of the input 1, input 2, and input 3.

[0178] When one input data is determined to be appropriate and two input data are determined to be inappropriate among three pieces of data input to input 1, input 2, and input 3, the control unit 1601 outputs, instead of the two pieces of data determined to be inappropriate, substitute data for each characteristic value determined based on the first characteristic value and the second characteristic value of the one piece of data determined to be appropriate as data for inference 1, data for inference 2, and data for inference 3, respectively, to the inference unit 1602. Also, when two input data are determined to be appropriate and one input data is determined to be inappropriate among the pieces of data input to input 1, input 2, and input 3, the control unit 1601 compares the characteristic values ​​of the two pieces of data determined to be appropriate and outputs substitute data determined based on the higher characteristic value as data for inference to the inference unit 1602, instead of the one piece of data determined to be inappropriate.

[0179] In addition, when there are multiple pieces of data judged to be inappropriate, the substitute data may be selected in descending order of the characteristic value from among the pieces of data judged to be appropriate, and the substitute data may be output as the number of pieces of data judged to be inappropriate as the data for inference. The method of controlling the data for inference will be described in detail later.

[0180] <Configuration of the inference unit 1602> Next, the configuration and functions of the inference unit 1602 of the fifth embodiment will be described with reference to FIG.

[0181] The difference from the first embodiment is that data for inference 3 has been added, and that a third layer 1701 corresponding to the data for inference 3 has been added to the inference model 1700. As for the other configurations, the same components as those in Fig. 2 of the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0182] The inference model 1700 has a first layer 211 that receives inference data 1 as input, a second layer 212 that receives inference data 2 as input, a third layer 1711 that receives inference data 3 as input, and a combination layer 1702 that combines the output of the first layer 211, the output of the second layer 212, and the output of the third layer 1701 to output the result of inferring the area of ​​interest.

[0183] <How to determine data for inference> Next, a method in which the control unit 1601 determines data for inference based on the suitability judgment result of the suitability judgment unit 1603 and the characteristic information of the characteristic information calculation unit 1604 will be described with reference to FIG.

[0184] Figure 18(a) shows examples of suitability assessment results when image 0 is input to input 1, image 1 is input to input 2, and sound 0 is input to input 3, and when image 0 is input to input 1, image 1 is input to input 2, and sound 2 is input to input 3.

[0185] The method for calculating the evaluation value of each input data is as described with reference to FIG.

[0186] In the fifth embodiment, the first threshold and the second threshold are set to two.

[0187] When image 0, image 1, and sound 0 are input, the evaluation value of each input data is 4 for image 0, 1 for image 1, and 3 for sound 0.

[0188] By comparing the evaluation value of each input data with the first threshold, image 0 and sound 0 are determined to be appropriate since they are equal to or greater than the first threshold, and image 1 is determined to be inappropriate since it is less than the first threshold.

[0189] By comparing the evaluation value of each input data with the second threshold, the highest evaluation value is image 0, which is 4, and the next highest evaluation value is sound 0, which is 3, so the difference between the evaluation value of image 0 and the evaluation value of sound 0 is 1. In this case, since the difference in evaluation value is less than the second threshold, all input data is determined to be appropriate. Suitability determination unit 1603 determines image 0 to be appropriate, image 1 to be inappropriate, and sound 0 to be appropriate based on the suitability determination result obtained by comparing the evaluation value of each input data with the first and second thresholds.

[0190] When image 0, image 1, and sound 2 are input, the evaluation value of each input data is 4 for image 0, 1 for image 1, and 2 for sound 2.

[0191] By comparing the evaluation value of each input data with the first threshold, image 0 and sound 2 are determined to be appropriate since they are equal to or greater than the first threshold, while image 1 is determined to be inappropriate since it is less than the first threshold.

[0192] Comparing the evaluation value of each input data with the second threshold, the highest evaluation value is 4 for Image 0, and the next highest evaluation value is 2 for Sound 2, so the difference between the evaluation values ​​of Image 0 and Sound 2 is 2. In this case, since the difference in the evaluation values ​​is greater than or equal to the second threshold, Image 0 is determined to be appropriate, and Image 1 and Sound 2 are determined to be inappropriate.

[0193] Although sound 2 is judged to be appropriate based on the first threshold, it is judged to be inappropriate based on comparison with the second threshold, so the suitability judgment unit 1603 gives priority to the inappropriate judgment result and judges image 0 to be appropriate, image 1 to be inappropriate, and sound 2 to be inappropriate.

[0194] Fig. 18(b) illustrates an example of substitute data determined based on the characteristic values ​​calculated for images 0 and 1. Fig. 18(c) illustrates an example of substitute data determined based on the characteristic values ​​calculated for sounds 0 and 2.

[0195] The control unit 1601 determines substitute data based on the characteristic values ​​of the input data in a manner similar to that shown in FIG. 6(a) of the first embodiment.

[0196] When image 0 is input, the characteristic information calculation unit 1604 calculates the characteristic value of the high sensitivity noise amount as 7 and the characteristic value of the blur amount as 8. The control unit 1601 determines the substitute data for each characteristic value as 2, as described in FIG.

[0197] When image 1 is input, the characteristic information calculation unit 1604 calculates the characteristic value of the high sensitivity noise amount as 4 and the characteristic value of the blur amount as 2. The control unit 1601 determines the substitute data for each characteristic value as 1, as described in FIG.

[0198] When sound 0 is input, the characteristic information calculation unit 1604 calculates the characteristic value of the noise sound to be 5 and the characteristic value of the volume to be 6. The control unit 1601 determines the substitute data for the noise sound to be 1 and the substitute data for the volume to be 2, as described in FIG.

[0199] When sound 2 is input, the characteristic information calculation unit 1604 calculates the characteristic value of the noise sound to be 2 and the characteristic value of the volume to be 4. The control unit 1601 determines the alternative data for the noise sound to be 0 and the alternative data for the volume to be 1, as described in FIG.

[0200] Figure 18(d) is a diagram explaining how the control unit 1601 determines inference data 1, inference data 2, and inference data 3 based on the suitability judgment result in Figure 18(a) and the alternative data determined in Figures 18(b) and 18(c).

[0201] 18(a), the suitability determination unit 1603 determines image 0 as appropriate, image 1 as inappropriate, and sound 0 as appropriate, and the control unit 1601 outputs image 0 determined to be appropriate as inference data 1, and sound 0 determined to be appropriate as inference data 3. Furthermore, the control unit 1601 outputs blur substitute data 2 having the highest characteristic value from among the characteristic values ​​of image 0 and sound 0 determined to be appropriate, as inference data 2.

[0202] When image 0 is input to input 1, image 1 to input 2, and sound 2 to input 3, as shown in Fig. 18(a), the suitability determination unit 1603 determines image 0 to be appropriate, image 1 to be inappropriate, and sound 2 to be inappropriate, and the control unit 1601 sets image 0, which has been determined to be appropriate, as data for inference 1. Furthermore, for data for inference 2 and data for inference 3, the control unit 1601 outputs alternative data for blur 2 and alternative data for high sensitivity noise 2 in descending order of characteristic values ​​of image 0, which has been determined to be appropriate.

[0203] In this embodiment, when there is a plurality of input data judged to be inappropriate, the input data judged to be appropriate are set as replacement data in descending order of characteristic values, however, the input data may be set to the same replacement data.

[0204] <Control processing> Next, the control process of the inference device 1600 of the fifth embodiment will be described with reference to FIG.

[0205] The processing in FIG. 19 is realized by the control unit 1601 executing a program stored in memory, and is started when image data and sound data are input to the inference device 1600.

[0206] In step S1901, the suitability determining unit 1603 calculates an evaluation value of the input data.

[0207] In step S1902, the suitability determination unit 1603 compares the evaluation value calculated in step S1901 with a first threshold. If the suitability determination unit 1603 determines that the evaluation value is equal to or greater than the first threshold, it determines that the input data is appropriate, and if the suitability determination unit 1603 determines that the evaluation value is less than the first threshold, it determines that the input data is inappropriate.

[0208] In step S1903, the suitability determination unit 1603 calculates the difference between the highest evaluation value and the second highest evaluation value for the evaluation values ​​calculated in step S1902, and compares the difference in evaluation values ​​with a second threshold. If the suitability determination unit 1603 determines that the difference in evaluation values ​​is equal to or greater than the second threshold, it determines the suitability of the input data according to the suitability determination result of step S1902. If the suitability determination unit 1603 determines that the difference in evaluation values ​​is less than the second threshold, it determines that all input data is appropriate, regardless of the suitability determination result of step S1902.

[0209] If the suitability judgment unit 1603 judges all the input data to be appropriate, the process proceeds to step S1904, and if there is any input data judged to be inappropriate, the process proceeds to step S1906. Note that in this embodiment, when one input data is judged to be appropriate and inappropriate in steps S1902 and S1903, the judgment of inappropriateness takes precedence, but this is not limited, and the input data may be judged to be appropriate.

[0210] In step S1904, the control unit 1601 outputs all the input data to the inference unit 1602 as data for inference.

[0211] In step S1905, the inference unit 1602 performs neural network calculations using the inference data output from the control unit 1601, outputs the inference result, and ends the process.

[0212] In step S1906, the control unit 1601 determines substitute data for each characteristic value based on the characteristic value for each piece of data determined to be appropriate.

[0213] In step S1907, the control unit 1601 outputs the alternative data determined in step S1906 as data for inference in place of the data determined to be appropriate and the data determined to be inappropriate, and proceeds to step S1905.

[0214] According to the fifth embodiment, the suitability of each input data is determined by comparing the evaluation value of each input data with a first threshold value, and the suitability of each input data is determined by comparing the difference between the highest evaluation value and the second highest evaluation value with a threshold value. As a result of the determination, data determined to be appropriate and alternative data determined based on the characteristic value of the data determined to be appropriate are determined as inference data to be output to the inference unit 1602 in place of the data determined to be inappropriate. As a result, when there are multiple pieces of data determined to be inappropriate in the multiple input data, it is possible to use the data determined to be appropriate and, for the data determined to be inappropriate, alternative data determined based on the characteristics of the appropriate data in the inference process, and the inference accuracy can be improved compared to the inference process using the data determined to be inappropriate.

[0215] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0216] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0217] The disclosure of this specification includes the following inference device, control method, and program. [Configuration 1] 1. An inference device, comprising: An inference means for executing an inference process using a plurality of inference data; a determination means for determining suitability of a plurality of input data input to the inference device for use in the inference process; a determination means for determining alternative data based on the input data determined to be appropriate by the determination means, instead of the input data determined to be inappropriate by the determination means; The inference device is characterized in that the inference means applies the input data determined to be appropriate and the alternative data to an inference model as the multiple inference data to perform inference processing. [Configuration 2] The determination means calculates evaluation information for each of the plurality of input data, The inference device according to configuration 1, characterized in that the suitability of the plurality of input data for use in the inference process is determined by comparing the evaluation information with a predetermined threshold value. [Configuration 3] The inference device described in configuration 2, characterized in that when the judgment means judges that all of the plurality of input data are appropriate, the decision means outputs the input data judged to be appropriate to the inference means as the plurality of inference data. [Configuration 4] The inference device according to configuration 2, wherein the decision means, when the judgment means judges all of the plurality of input data to be inappropriate, appropriately changes the judgment result of the input data with the highest evaluation information. [Configuration 5] A calculation means for calculating characteristic information for each of the plurality of input data, 5. The inference device according to claim 2, wherein the determining means determines the substitute data based on characteristic information for each of the input data. [Configuration 6] The calculation means calculates a plurality of pieces of characteristic information for each of the plurality of input data, The inference device according to configuration 5, wherein the determining means determines the alternative data based on a plurality of pieces of characteristic information for each of the input data. [Configuration 7] 7. The inference device according to configuration 5 or 6, wherein the characteristic information is a value obtained by normalizing the characteristic acquired by the calculation means for each of the input data. [Configuration 8] 8. The inference device according to any one of configurations 5 to 7, wherein the plurality of input data includes image data and sound data. [Configuration 9] When the input data is image data, the evaluation information and the characteristic information include a value related to any one of the amount of blur, brightness, and noise corresponding to sensitivity at the time of capturing the image data, 9. The inference apparatus according to claim 8, wherein, when the input data is sound data, the input data includes a value relating to a volume or a noise sound. [Configuration 10] When the input data is the image data, the characteristic information includes first characteristic information on a noise amount corresponding to a sensitivity at the time of capturing the image data, and second characteristic information on a blur amount of the image data, When the input data is sound data, the input data includes first characteristic information on a noise sound and second characteristic information on a volume of the noise sound, 10. The inference device according to claim 9, wherein the determining means determines the alternative data based on the first characteristic information and the second characteristic information for each of the input data. [Configuration 11] The plurality of input data include a plurality of image data each having a different orientation of a subject, the evaluation information is a value corresponding to a subject orientation for each image data; The inference device according to any one of configurations 5 to 7, wherein the characteristic information is a value relating to the amount of noise at the time of shooting for each image data. [Configuration 12] The plurality of input data include a plurality of image data captured successively, the evaluation information is a value corresponding to a luminance difference between subsequent image data and previous image data in a time series; The inference device according to any one of configurations 5 to 7, wherein the characteristic information is a value relating to the amount of noise at the time of shooting for each image data. [Configuration 13] 8. The inference device according to any one of configurations 5 to 7, wherein the plurality of input data includes three or more types of data including image data and sound data. [Configuration 14] The inference device described in configuration 13, characterized in that the judgment means judges the suitability of the multiple input data by comparing evaluation information calculated for each input data with a first threshold value. [Configuration 15] The inference device described in configuration 14, characterized in that the determination means calculates the difference between the highest evaluation information and the next highest evaluation information, and if it determines that the difference is greater than or equal to a second threshold, determines the suitability of the input data according to the judgment result based on the first threshold, and if it determines that the difference is less than the second threshold, determines that all input data is appropriate regardless of the judgment result based on the first threshold. [Configuration 16] The inference means executes an inference process using a trained inference model and parameters, An inference device described in any one of configurations 1 to 15, characterized in that the inference model is subjected to a learning process in advance using learning data including input data for learning and teacher data. [Configuration 17] The inference device according to configuration 16, characterized in that the learning input data includes a data set that combines a plurality of input data that the judgment means judges to be appropriate, or a data set that combines input data that the judgment means judges to be appropriate with alternative data that is determined based on characteristic information of the input data. [Configuration 18] The inference model is a neural network, The inference device according to any one of configurations 1 to 17, wherein the inference process is deep learning using a neural network. [Configuration 19] A method for controlling an inference device that executes an inference process using a plurality of inference data, comprising: a determination step of determining suitability of a plurality of input data input to the inference device for use in the inference process; a determining step of determining alternative data based on the input data determined to be appropriate in the determining step, instead of the input data determined to be inappropriate in the determining step; A control method characterized by having an inference step of applying the input data determined to be appropriate and the alternative data as the multiple inference data to an inference model to perform inference processing. [Configuration 20] A program for causing a computer to function as the inference device according to any one of claims 1 to 18. [Explanation of symbols]

[0218] 100, 1600... inference device, 101, 1601... control unit, 102, 1602... inference unit, 103, 1603... suitability determination unit, 104, 1604... characteristic information calculation unit

Claims

1. 1. An inference device, comprising: An inference means for executing an inference process using a plurality of inference data; a determination means for determining suitability of a plurality of input data input to the inference device for use in the inference process; a determination means for determining alternative data based on the input data determined to be appropriate by the determination means, instead of the input data determined to be inappropriate by the determination means; The inference device is characterized in that the inference means applies the input data determined to be appropriate and the alternative data to an inference model as the multiple inference data to execute an inference process.

2. The determination means calculates evaluation information for each of the plurality of input data, The inference device according to claim 1 , wherein the suitability of the plurality of input data for use in the inference process is determined by comparing the evaluation information with a predetermined threshold value.

3. The inference device according to claim 2, characterized in that, when the judgment means judges that all of the plurality of input data are appropriate, the decision means outputs the input data judged to be appropriate to the inference means as the plurality of inference data.

4. 3. The inference device according to claim 2, wherein the determining means changes the judgment result of the input data with the highest evaluation information to an appropriate one when the judging means judges all of the plurality of input data to be inappropriate.

5. A calculation means for calculating characteristic information for each of the plurality of input data, 3. The inference device according to claim 2, wherein the determining means determines the substitute data on the basis of characteristic information for each of the input data.

6. The calculation means calculates a plurality of pieces of characteristic information for each of the plurality of input data, 6. The inference device according to claim 5, wherein said determining means determines said substitute data on the basis of a plurality of pieces of characteristic information for each of said input data.

7. 6. The inference device according to claim 5, wherein the characteristic information is a normalized value of the characteristic obtained by the calculation means for each of the input data.

8. 6. The inference device according to claim 5, wherein the plurality of input data include image data and sound data.

9. When the input data is image data, the evaluation information and the characteristic information include a value related to any one of the amount of blur, brightness, and noise corresponding to sensitivity at the time of capturing the image data, 9. The inference apparatus according to claim 8, wherein the input data, when said input data is sound data, includes a value relating to a volume or a noise sound.

10. When the input data is the image data, the characteristic information includes first characteristic information on a noise amount corresponding to a sensitivity at the time of capturing the image data, and second characteristic information on a blur amount of the image data, When the input data is sound data, the input data includes first characteristic information on a noise sound and second characteristic information on a volume of the noise sound, 10. The inference device according to claim 9, wherein the determining means determines the substitute data based on the first characteristic information and the second characteristic information for each of the input data.

11. The plurality of input data include a plurality of image data each having a different orientation of a subject, the evaluation information is a value corresponding to a subject orientation for each image data; 6. The inference device according to claim 5, wherein the characteristic information is a value related to the amount of noise at the time of shooting for each image data.

12. The plurality of input data include a plurality of image data captured successively, the evaluation information is a value corresponding to a luminance difference between subsequent image data and previous image data in a time series; 6. The inference device according to claim 5, wherein the characteristic information is a value related to the amount of noise at the time of shooting for each image data.

13. 6. The inference device according to claim 5, wherein the plurality of input data include three or more types of data including image data and sound data.

14. 14. The inference device according to claim 13, wherein the determination means determines suitability of the plurality of input data by comparing evaluation information calculated for each of the plurality of input data with a first threshold value.

15. The inference device described in claim 14, characterized in that the determination means calculates the difference between the highest evaluation information and the next highest evaluation information, and if it determines that the difference is greater than or equal to a second threshold, determines the suitability of the input data in accordance with the judgment result based on the first threshold, and if it determines that the difference is less than the second threshold, determines that all input data is appropriate regardless of the judgment result based on the first threshold.

16. The inference means executes an inference process using a trained inference model and parameters, The inference device according to claim 1 , characterized in that the inference model is pre-trained using training data including input data for learning and teacher data.

17. The inference device according to claim 16, characterized in that the learning input data includes a data set that combines a plurality of input data that the judgment means judges to be appropriate, or a data set that combines input data that the judgment means judges to be appropriate with alternative data that is determined based on characteristic information of the input data.

18. The inference model is a neural network, The inference device according to claim 1 , wherein the inference process is deep learning using a neural network.

19. A method for controlling an inference device that executes an inference process using a plurality of inference data, comprising: a determination step of determining suitability of a plurality of input data input to the inference device for use in the inference process; a determining step of determining alternative data based on the input data determined to be appropriate in the determining step, instead of the input data determined to be inappropriate in the determining step; A control method characterized by having an inference step of applying the input data determined to be appropriate and the alternative data as the multiple inference data to an inference model to perform inference processing.

20. A program for causing a computer to function as the inference device according to any one of claims 1 to 18.

Images