Inference apparatus, control method, and program

Abstract

To shorten the time of inference processing by improving inference accuracy even when data unsuitable for learning are included in a plurality of kinds of input data.SOLUTION: An inference apparatus includes an inference unit configured to execute inference processing using a plurality of types of input data and auxiliary data for each of the input data, a determination unit configured to determine suitability of the plurality of types of input data input to the inference apparatus for use in the inference processing, a calculation unit configured to calculate characteristic information for each of the plurality of types of input data, and a control unit configured to determine the auxiliary data of the input data based on the suitability and the characteristic information of the input data. The inference means replaces input data determined to be inappropriate by the determination means among the plurality of kinds of input data and auxiliary data for each input data with auxiliary data determined on the basis of characteristic information of input data determined to be appropriate by the determination means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for inputting a plurality of data and executing inference processing. [Background technology]

[0002] Recently, AI (artificial intelligence) has been used in image processing to recognize people's faces and expressions from photographed images. In image processing using AI, photographed images, etc., are used as input data and applied to a trained inference model, where inference processing is performed using machine learning, and the inference results are output.

[0003] In this case, if the data used in the inference process is a single type of input data (single-modal data), it is not sufficient to improve the inference accuracy. Therefore, Patent Document 1 describes a technology in which a learning process for an inference model is performed using multiple types of input data (multi-modal data), thereby improving the inference accuracy more than when learning process is performed using single-modal data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-2023 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, if the multiple types of input data include inappropriate data that is not suitable for learning processing, it becomes difficult to improve the inference accuracy in inference processing using an inference model trained with such inappropriate data.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that improves inference accuracy and shortens the inference processing time even when multiple types of input data include data that is not suitable for learning. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the inference device of the present invention comprises an inference means for performing an inference process using multiple types of input data and auxiliary data for each of the input data, a judgment means for judging the suitability of multiple types of input data input to the inference device for use in the inference process, a calculation means for calculating characteristic information for each of the multiple types of input data, and a control means for determining auxiliary data for the input data based on the suitability and characteristic information of the input data, and the inference means replaces input data determined to be inappropriate by the judgment means among the multiple types of input data and the auxiliary data for each of the input data with auxiliary data determined based on the characteristic information of the input data determined to be appropriate by the judgment means. [Effects of the Invention]

[0008] According to the present invention, even when multiple types of input data include data that is not suitable for learning, it is possible to improve inference accuracy and shorten the time required for inference processing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of an inference device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating the configuration of an inference unit according to the present embodiment. [Figure 3] 3A and 3B are diagrams illustrating image data input to the inference device of the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating sound data input to the inference device of the present embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of an aptitude determination result according to the present embodiment. [Figure 6] 5A to 5C are diagrams for explaining a method for calculating characteristic information of input data and a method for determining auxiliary data according to the present embodiment. [Figure 7] FIG. 3 is a diagram illustrating inference data according to the present embodiment. [Figure 8] 3 is a flowchart showing a control process according to the present embodiment. [Figure 9] FIG. 2 is a diagram illustrating the learning process of the inference model of this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] <Device configuration> First, the configuration and functions of an inference device 100 of this embodiment will be described with reference to FIG.

[0012] The inference device 100 has a control unit 101 and an inference unit 102, and performs inference processing using a trained inference model and parameters. The inference device 100 applies multiple types of input data to the inference model to perform multimodal inference processing and output the inference results.

[0013] In this embodiment, the inference device 100 receives image data as a first input and audio data as a second input, and outputs an area focusing on people having a conversation in a captured image as an inference result.

[0014] In the following description, the first input is image data, the second input is sound data, and the output is a region of interest.

[0015] The inference unit 102 applies first inference data, first auxiliary data related to the first inference data, second inference data, and second auxiliary data related to the second inference data to an inference model, which will be described later in Fig. 2, to perform inference processing using deep learning, and outputs the inference result. The inference result may be output to internal storage of the inference device 100, or may be output to an external device via a wide area network such as the Internet.

[0016] The suitability determination unit 103 determines the suitability of image data and sound data as input data when the inference unit 102 performs inference processing. The suitability determination unit 103 calculates evaluation information for the image data and sound data, and compares the evaluation information with a threshold. The suitability determination unit 103 determines that input data whose evaluation information is equal to or greater than the threshold is suitable for inference processing (hereinafter referred to as "suitable"), and determines that input data whose evaluation information is less than the threshold is not suitable for inference processing (hereinafter referred to as "inappropriate"), and outputs the determination result to the control unit 101. The method for determining the suitability of input data will be described in detail later.

[0017] The characteristic information calculation unit 104 acquires the characteristics of the image data and sound data, and calculates normalized characteristic values ​​according to the characteristics. The method for calculating the characteristic values ​​will be described in detail later.

[0018] The control unit 101 has a processor such as a CPU, and memories such as a ROM and a RAM. The control unit 101 controls the suitability determination unit 103, the characteristic information calculation unit 104, and the inference unit 102 by executing programs stored in the memory. The control unit 101 controls auxiliary data related to the inference data to be output to the inference unit 102 based on characteristic values ​​of the image data and sound data. The control unit 101 determines auxiliary data related to the inference data based on the characteristic values ​​of the image data and sound data and the determination result by the suitability determination unit 103. If the suitability determination unit 103 determines that the image data and sound data are appropriate, the control unit 101 outputs the image data as first inference data and the sound data as second inference data to the inference unit 102, and outputs predetermined fixed values ​​(e.g., 0 (zero)) to the inference unit 102 as first auxiliary data related to the first inference data and second auxiliary data related to the second inference data. Furthermore, when the suitability determination unit 103 determines that the image data is appropriate and the sound data is inappropriate, the control unit 101 outputs the image data as first inference data and the sound data as second inference data to the inference unit 102, and outputs a predetermined fixed value (e.g., 0) as first auxiliary data related to the first inference data and predetermined substitute data (e.g., image auxiliary data described later in FIG. 7) as second auxiliary data related to the second inference data to the inference unit 102. When the suitability determination unit 103 determines that the image data is inappropriate and the sound data is appropriate, the control unit 101 outputs the image data as first inference data and the sound data as second inference data to the inference unit 102, and outputs predetermined substitute data (e.g., sound auxiliary data described later in FIG. 7) as first auxiliary data related to the first inference data and a predetermined fixed value (e.g., 0) as second auxiliary data related to the second inference data to the inference unit 102. Details of a method for controlling auxiliary data will be described later.

[0019] <Configuration of inference unit 102> Next, the configuration and functions of the inference unit 102 of this embodiment will be described with reference to FIG.

[0020] The inference unit 102 has an inference model 200. The inference model 200 is configured by a neural network, and in this embodiment, is configured by a convolutional neural network (CNN). Note that the inference model in this embodiment is not limited to a CNN, and may be configured by a neural network such as an RNN (Recurrent Neural Network) or a fully connected type. The calculation method of a neural network is well known, so a description thereof will be omitted.

[0021] The inference processing by deep learning in the inference unit 102 and the learning processing of the inference model 200, which will be described later with reference to FIG. 9, can be executed by a GPU or a DSP (Digital Signal Processor). A GPU or DSP is a processor capable of performing a large amount of product-sum operations, bias addition, nonlinear processing, etc., and has the computational processing power to perform matrix operations of neural networks in a short period of time. Note that the inference processing and learning processing may be performed by a CPU in the control unit 101 and a GPU or DSP in the inference unit 102 working together, or may be performed by either the CPU in the control unit 101 or the GPU or DSP in the inference unit 102.

[0022] The inference model 200 has a first input layer 211 that receives as input first inference data, which is image data, and a second input layer 212 that receives as input second inference data, which is audio data. The inference model 200 also has a first hidden layer 213 that receives as input the output of the first input layer 211 and first auxiliary data related to the first inference data output by the control unit 101, and a second hidden layer 214 that receives as input the output of the second input layer 212 and second auxiliary data related to the second inference data output by the control unit 101. The inference model 200 also has a combination layer 210 that combines the output of the first hidden layer 213 and the output of the second hidden layer 214 to output the result of inferring the region of interest.

[0023] The first input layer 211 extracts features related to the first inference data and outputs them to the first hidden layer 213. The first input layer 211 extracts the likelihood of the data being a person of interest obtained from an image, for example.

[0024] The second input layer 212 extracts features related to the second inference data and outputs them to the second hidden layer 214. The second input layer 212 extracts the likelihood of the person being a person of interest obtained from, for example, speech.

[0025] The first hidden layer 213 receives as input the feature values ​​output from the first input layer 211 and the first auxiliary data. If the first auxiliary data is a predetermined fixed value (e.g., 0), the first hidden layer 213 examines the accuracy of the output of the first input layer 211 and outputs the feature values ​​output from the first input layer 211 to the subsequent combined layer 210. If the first auxiliary data is not a predetermined fixed value (e.g., 0) but is predetermined alternative data, the first hidden layer 213 extracts feature values ​​related to the first auxiliary data and outputs them to the combined layer 210.

[0026] The second hidden layer 214 receives as input the feature values ​​output from the second input layer 212 and the second auxiliary data. If the second auxiliary data is a predetermined fixed value (e.g., 0), the second hidden layer 214 examines the accuracy of the output from the second input layer 212 and outputs the feature values ​​output from the second input layer 212 to the subsequent combined layer 210. If the second auxiliary data is not a predetermined fixed value but predetermined alternative data, the second hidden layer 214 extracts feature values ​​related to the second auxiliary data and outputs them to the combined layer 210.

[0027] The combined layer 210 receives the output of the first intermediate layer 213 and the output of the second intermediate layer 214, combines the features output from the first intermediate layer 213 and the features output from the second intermediate layer 214, and outputs an inference result.

[0028] Although the present embodiment illustrates an example in which there are two intermediate layers, the number of intermediate layers is not limited to two, and the number of intermediate layers may be changed depending on the time required for the aptitude determination process and the characteristic information calculation process, etc. Furthermore, the auxiliary data to be input for each intermediate layer may be changed.

[0029] <Explanation of image data> Next, with reference to FIG. 3, the image data input to the inference device 100 of this embodiment will be described.

[0030] FIG. 3(a) shows an example of image 0, which includes a talking person and a walking person, has no blur, and has some image noise.

[0031] FIG. 3(b) illustrates an example of an image 1 that is more blurred and noisy than the image 0.

[0032] The image noise in this embodiment is high-sensitivity noise that occurs due to high ISO sensitivity when capturing an image, but is not limited to this and may be long-term noise that occurs when the image sensor heats up.

[0033] In this embodiment, the blurring of an image refers to blurring caused by camera shake or subject movement when capturing an image, or blurring caused by focus shift, but is not limited to these and may also be blurring caused by the optical system when capturing an image.

[0034] The method for calculating the amount of noise and the amount of blur is well known, and therefore the explanation will be omitted.

[0035] <Sound data explanation> Next, with reference to FIG. 4, the sound data input to the inference device 100 of this embodiment will be described.

[0036] FIG. 4(a) illustrates an example of the waveform of sound 0, which is the voice of a person speaking, among the sound data input to the inference device 100.

[0037] FIG. 4(b) illustrates an example of a waveform of noise that is generated around the inference device 100 and does not include wind noise or other noise.

[0038] FIG. 4(c) illustrates an example of a sound waveform of sound 1, in which the volume of speech of the person speaking is small and the amplitude of the sound is smaller than sound 0, in comparison with sound 0 in FIG. 4(a).

[0039] Figure 4(d) illustrates the noise sound of Sound 1, which is an example of the waveform of the noise sound of Sound 1, which is louder than Sound 0 due to environmental sounds occurring around the inference device 100, such as the sound of a car driving.

[0040] The method for calculating the volume and noise amount is well known, and therefore the explanation will be omitted.

[0041] <Results of input data suitability assessment> Next, a method for determining the suitability of image data and sound data by the suitability determining unit 103 of this embodiment will be described with reference to FIG.

[0042] The suitability determination unit 103 calculates evaluation information for determining the suitability of image data and sound data as input data within a range of evaluation values ​​from a minimum of 0 to a maximum of 4. In this embodiment, the threshold value is set to 2 to determine the suitability of the input data. The suitability determination unit 103 determines that an evaluation value of 2 or more is appropriate, and that an evaluation value of less than 2 is unusable.

[0043] In this embodiment, the larger the evaluation value, the more suitable the data is for inference processing, and the smaller the evaluation value, the less suitable the data is for inference processing, and appropriate data and inappropriate data are determined based on threshold value 2. Note that the method for determining the suitability of input data is not limited to the above-mentioned method, and any method may be used.

[0044] The suitability determination unit 103 determines the suitability of input image data based on the amount of blur, and determines the suitability of input sound data based on the volume.

[0045] Here, a method for calculating an evaluation value relating to the amount of blur of image data and an evaluation value relating to the volume of sound data will be described.

[0046] First, a method for calculating an evaluation value of the amount of blur of image data will be described.

[0047] In this embodiment, the object in the image is recognized to identify the person speaking and acquire the area of ​​interest. Therefore, the greater the amount of blur in the image data, the more difficult it becomes to identify the person speaking, so the amount of blur in the image data is used as an evaluation value to determine suitability. For example, if the amount of blur in image data that is so blurred that the person cannot be identified is set to 100%, and the amount of blur in image data with no blur is set to 0%, the greater the amount of blur, the lower the evaluation value is set, and the less the amount of blur, the higher the evaluation value is set. In this embodiment, the evaluation value is set to 0 to 4 for every 20% of the amount of blur.

[0048] The threshold value of the amount of blur in this embodiment is a threshold value for determining whether or not a person can be identified.

[0049] Next, a method for calculating an evaluation value of the volume of sound data will be described.

[0050] In this embodiment, when recognizing a subject in an image, it is difficult to capture the voices of people speaking if the volume is low, so suitability is determined using the volume of the sound data as an evaluation value. For example, if the volume of sound data with no conversation volume is set to 0% and the limit value of the volume of a sound collection device such as a microphone that captures sound is set to 100%, the lower the volume, the lower the evaluation value is set, and the higher the volume, the higher the evaluation value is set. In this embodiment, the evaluation value is set to 0 to 4 for every 20% of the volume. Furthermore, the volume threshold in this embodiment is a threshold for determining whether the voices of people speaking can be identified. Note that if the volume is too loud and the conversation cannot be identified, the evaluation value may be lowered.

[0051] In this embodiment, the evaluation value is calculated based on the amount of blur in the image data and the volume of the audio data, and the suitability is determined based on the evaluation value, but the evaluation value may also be based on brightness, high-sensitivity noise, or noise such as ambient noise. Note that the method for calculating the evaluation value is well known, so a description thereof will be omitted.

[0052] FIG. 5(a) illustrates the results of determining the suitability of each of the input data images 0 and 1 based on the evaluation value calculated from the amount of blur and the threshold value.

[0053] When image 0 is input, the suitability determination unit 103 determines that image 0 is appropriate because there is no blur and the evaluation value of the amount of blur is 4, which is greater than or equal to threshold 2. When image 1 is input, the suitability determination unit 103 determines that image 1 is appropriate because there is a lot of blur and the evaluation value is 1, which is less than threshold 2.

[0054] FIG. 5(b) illustrates the results of determining the suitability of each of sound 0 and sound 1 as input data based on the volume and threshold value.

[0055] When sound 0 is input, the suitability determination unit 103 determines that the sound is appropriate by setting the evaluation value of the volume to 3, which is equal to or greater than threshold value 2. When sound 1 is input, the suitability determination unit 103 determines that the sound is inappropriate by setting the evaluation value to 0, which is less than threshold value 2, because the sound waveform cannot be obtained.

[0056] <How to determine input data characteristic values ​​and auxiliary data> Next, a method for calculating characteristic information of input data and a method for determining auxiliary data will be described with reference to FIG.

[0057] 6(a) illustrates an example of characteristic values ​​of input data and auxiliary data corresponding to the characteristic values. The characteristic information calculation unit 104 calculates the characteristic values, assuming that the characteristics obtainable from the input data range from a minimum of 0 to a maximum of 9. The control unit 101 determines auxiliary data 2 when the characteristic value is 6 to 9, auxiliary data 1 when the characteristic value is 3 to 5, and auxiliary data 0 when the characteristic value is 0 to 2.

[0058] In this embodiment, the characteristics of the high-sensitivity noise amount in the image data are acquired, and a normalized characteristic value is calculated.

[0059] Here, a normalization method according to the characteristics of the high-sensitivity noise amount in the image data will be described.

[0060] The more high-sensitivity noise there is in the image data, the more difficult it becomes to identify the person in conversation as the person of interest. For example, if the amount of noise in image data with so much high-sensitivity noise that it is impossible to identify the person is set to 100%, and the amount of noise in an image with no noise is set to 0%, the greater the amount of high-sensitivity noise, the lower the characteristic value is set, and the less the amount of high-sensitivity noise, the higher the characteristic value is set. In this embodiment, the characteristic value is calculated as 0 to 9 for every 10% of the noise amount.

[0061] 6(b) illustrates the characteristic values ​​of high-sensitivity noise in images 0 and 1 when images 0 and 1 in FIG. 3 are used as input data, and auxiliary data determined based on the characteristic values. Note that, in this embodiment, high-sensitivity noise is illustrated as an example of the characteristic value calculated by the characteristic information calculation unit 104, but the characteristic value may also be the S / N ratio, brightness, amount of blur, etc. in the image data.

[0062] The information used for calculation by the aptitude determination unit 103 and the characteristic information calculation unit 104 may be the same. In this case, the information is the amount of blurring and high-sensitivity noise in this embodiment. For example, in this embodiment, both the aptitude determination unit 103 and the characteristic information calculation unit 104 may calculate the evaluation value and the characteristic value using the amount of blurring.

[0063] When image 0 is input, the characteristic information calculation unit 104 calculates 7 as the characteristic value of high sensitivity noise, and the control unit 101 determines the auxiliary data to be 2 from Fig. 6(a) since the characteristic values ​​are 6 to 9. When image 1 is input, the characteristic information calculation unit 104 calculates 4 as the characteristic value of high sensitivity noise, and the control unit 101 determines the auxiliary data to be 1 from Fig. 6(a) since the characteristic values ​​are 3 to 5.

[0064] Figure 6(c) illustrates the characteristic values ​​of noise sounds, which are environmental sounds around the inference device 100, when sound 0 and sound 1 in Figure 4 are used as input data, and the auxiliary data determined by the control unit 101 based on the characteristic values.

[0065] When sound 0 is input, the characteristic information calculation unit 104 calculates 5 as the characteristic value of the noise sound, and the control unit 101 determines the auxiliary data to be 1 from Fig. 6(a) because the characteristic value is 3 to 5. When sound 1 is input, the characteristic information calculation unit 104 calculates 2 as the characteristic value of the noise sound, and the control unit 101 determines the auxiliary data to be 0 from Fig. 6(a) because the characteristic value is 0 to 2.

[0066] The normalization method is not limited to the above-mentioned method, and any method that can calculate a quantified characteristic value based on a certain standard may be used. In this embodiment, the characteristic value is set to 10 levels from 0 to 9, and three types of auxiliary data are set, but the normalization method is not limited to this.

[0067] Furthermore, the characteristic information calculation unit 104 may calculate characteristic values ​​only for data that has been determined to be appropriate by the appropriateness determination unit 103. This makes it possible to reduce the time required for the calculation process of the characteristic information calculation unit 104.

[0068] <How to determine data for inference> Next, referring to Figure 7, we will explain a method for determining auxiliary data to be output from the control unit 101 to the inference unit 102 based on the characteristic values ​​of image data and sound data as input data and the suitability judgment result of the suitability judgment unit 103.

[0069] In FIG. 7, when an image of 0 is input as the first input and an audio of 0 is input as the second input, both input data are appropriate, so the control unit 101 determines the first auxiliary data and the second auxiliary data to be predetermined fixed values ​​(for example, 0).

[0070] When image 0 is input as the first input and sound 1 is input as the second input, image 0 is appropriate but sound 1 is inappropriate, so the control unit 101 determines the first auxiliary data to be a predetermined fixed value (for example, 0) and determines the second auxiliary data to be predetermined substitute data. In this case, the predetermined substitute data becomes image auxiliary data 2 in Figure 6(b) corresponding to the characteristic value of image 0, which is appropriate as the first input.

[0071] When image 1 is input as the first input and sound 0 is input as the second input, image 1 is inappropriate but sound 0 is appropriate, so the control unit 101 determines the second auxiliary data to be a predetermined fixed value (for example, 0) and determines the first auxiliary data to be predetermined alternative data. In this case, the predetermined alternative data becomes sound auxiliary data 1 in Figure 6(c) that corresponds to the characteristic value of sound 0, which is appropriate as the second input.

[0072] If image 1 is input as the first input and sound 1 as the second input, both input data are inappropriate, so the suitability determination unit 103 again compares the evaluation values ​​used to determine the suitability of image 1 and sound 1, and appropriately changes the suitability determination result of image 1, which has the higher evaluation value. Then, the control unit 101 determines the first auxiliary data to be a predetermined fixed value (e.g., 0), and determines the second auxiliary data to be predetermined substitute data. In this case, the predetermined substitute data becomes image auxiliary data 1 in Figure 6(b) that corresponds to the characteristic value of image 1, which is appropriate as the first input.

[0073] The predetermined fixed value is not limited to 0, but may be a predetermined value or NULL.

[0074] <Control processing of inference data> Next, the auxiliary data control process performed by the inference device 100 of this embodiment will be described with reference to FIG.

[0075] The processing of Figure 8 is realized by the control unit 101 executing a program stored in memory and controlling the suitability judgment unit 103, characteristic information calculation unit 104 and inference unit 102, and is initiated when image data and sound data are input to the inference device 100.

[0076] In step S801, the suitability determining unit 103 calculates an evaluation value of the input data.

[0077] In step S802, the suitability determination unit 103 compares the evaluation value calculated in step S801 with a threshold. If the suitability determination unit 103 determines that the evaluation value is equal to or greater than the threshold, it determines the input data to be appropriate, and if it determines that the evaluation value is less than the threshold, it determines the input data to be inappropriate. If the suitability determination unit 103 determines that all of the input data is appropriate, it proceeds to step S803. If the suitability determination unit 103 determines that some of the input data is inappropriate, it proceeds to step S805, and if it determines that all of the input data is inappropriate, it proceeds to step S807.

[0078] In step S803, since the suitability determination unit 103 has determined that the image data and sound data are appropriate, the control unit 101 outputs a predetermined fixed value (e.g., 0) to the inference unit 102 as the input data as first inference data and second inference data, first auxiliary data regarding the first inference data, and second auxiliary data regarding the second inference data.

[0079] In step S804, the inference unit 102 performs inference processing using the first inference data and the second inference data, and the first auxiliary data and the second auxiliary data output from the control unit 101, outputs the inference result, and terminates the processing.

[0080] In step S805, the control unit 101 determines auxiliary data for the input data determined to be appropriate in step S802 and auxiliary data for the input data determined to be inappropriate in step S802.

[0081] In step S806, the control unit 101 outputs the first inference data, the second inference data, and the auxiliary data for the input data determined in step S805 to the inference unit 102, and the process proceeds to step S804.

[0082] In step S804, the inference unit 102 performs inference processing using the inference data determined to be appropriate output from the first intermediate layer 213 or the second intermediate layer 214 and auxiliary data that replaces the inference data determined to be inappropriate, outputs the inference result, and terminates the processing.

[0083] In step S807, the suitability judgment unit 103 appropriately changes the suitability judgment result of the input data with the highest evaluation value among the input data judged to be inappropriate in step S802, and proceeds to steps S805, S806, and S804, before terminating the process.

[0084] According to this embodiment, when performing inference processing using input data and auxiliary data, the suitability of the input data is judged, and input data judged to be inappropriate in the inference processing is replaced with auxiliary data determined based on the characteristic values ​​of the input data judged to be appropriate, thereby making it possible to shorten the time required for the inference processing compared to replacing the input data with auxiliary data before performing the inference processing.

[0085] Furthermore, if a portion of the input data is determined to be inappropriate, the inference process can improve the inference accuracy compared to an inference process that uses input data determined to be inappropriate by replacing the input data determined to be inappropriate with auxiliary data determined based on the characteristic values ​​of the input data determined to be appropriate.

[0086] Furthermore, if all input data is determined to be appropriate, the input data determined to be appropriate is used as inference data, thereby preventing a decrease in inference accuracy.

[0087] Furthermore, if all input data is judged to be inappropriate, the suitability of the input data with the highest evaluation value among the input data judged to be inappropriate is appropriately changed, and in the inference process, the other input data judged to be inappropriate is replaced with auxiliary data determined based on the characteristic value of the input data with the highest evaluation value, thereby improving the inference accuracy compared to inference processing using input data that is all judged to be inappropriate.

[0088] <Learning process for inference model> Next, the learning process of the inference model 200 of FIG. 2 will be described with reference to FIG.

[0089] The inference model 200 of the inference unit 102 of this embodiment has undergone a learning process in advance by an information processing device such as a personal computer (PC) different from the inference device 100.

[0090] The learning process of the inference model 200 can be the same as that of the inference unit 102 of the inference device 100.

[0091] The learning process of the inference model 200 in this embodiment is a process of optimizing the parameters of the inference model 200 using previously prepared learning data and updating the parameters of the inference model 200 to the optimized parameters. The learning data includes input data and training data. The input data is input to the inference model 200, and the parameter adjustment unit 300 optimizes the parameters of the inference model 200 so that the deviation between the output data of the inference model 200 and the training data is minimized. The inference model 200 obtains the optimized parameters from the parameter adjustment unit 300 and updates the existing parameters.

[0092] The learning data includes learning input data and teacher data. The learning input data is a dataset prepared by combining multiple data, and in this embodiment, includes first inference data, first auxiliary data related to the first inference data, second inference data, and second auxiliary data related to the second inference data. The first dataset is a dataset combining image data and sound data determined to be appropriate by the aptitude determination unit 103, and their auxiliary data, which are 0 or NULL. The second dataset is a dataset combining image data determined to be appropriate by the aptitude determination unit 103 and image auxiliary data determined based on characteristic values ​​of the image data determined to be appropriate by the aptitude determination unit 103 in place of sound data determined to be inappropriate by the aptitude determination unit 103. The third dataset is a dataset combining sound data determined to be appropriate by the aptitude determination unit 103 and sound auxiliary data determined based on characteristic values ​​of the sound data determined to be appropriate by the aptitude determination unit 103 in place of image data determined to be inappropriate by the aptitude determination unit 103.

[0093] The training data is output data output by a preliminary training process using training input data, and in this embodiment, it is data of a region of interest output by a training process using image data and sound data as inference data as input data.

[0094] By using the above learning data in the learning process of the inference model 200, the learning process is limited to input data that is a combination of input data and auxiliary data determined based on the characteristic values ​​of the input data, making it possible to reduce the number of parameters and the amount of data.

[0095] The input data for learning is not limited to image data or sound data, but may be any data that can provide evaluation values ​​and characteristic values ​​of the data.

[0096] Furthermore, the input data for learning is not limited to two, the first inference data and the second inference data, but may be three or more input data.In this case, the inference model 200 may have processing layers corresponding to multiple types of input data and may perform multiple learning processes.

[0097] Note that since the learning algorithm of the neural network, such as the backpropagation method, is well known, a description thereof will be omitted.

[0098] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0099] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0100] The disclosure of this specification includes the following inference device, control method, and program. [Item 1] 1. An inference device, comprising: an inference means for executing an inference process using a plurality of types of input data and auxiliary data for each of the input data; a determination means for determining suitability of a plurality of types of input data input to the inference device for use in the inference process; a calculation means for calculating characteristic information for each of the plurality of types of input data; a control means for determining auxiliary data for the input data based on the suitability of the input data and characteristic information; The inference device is characterized in that the inference means replaces input data that is determined to be inappropriate by the judgment means, among the multiple types of input data and auxiliary data for each of the input data, with auxiliary data determined based on characteristic information of the input data that is determined to be appropriate by the judgment means. [Item 2] the inference means executes an inference process by applying the plurality of types of input data and auxiliary data for each of the input data to an inference model; The inference device described in item 1, characterized in that the inference model includes an input layer to which the multiple types of input data are input, and an intermediate layer to which auxiliary data for each of the input data is input. [Item 3] The inference device described in item 2 is characterized in that the intermediate layer outputs input data determined to be appropriate by the judgment means to a subsequent stage, and outputs auxiliary data determined based on characteristic information of the input data determined to be appropriate by the judgment means to a subsequent stage in place of input data determined to be inappropriate by the judgment means. [Item 4] 4. The inference device according to item 2 or 3, wherein the number of intermediate layers is changed depending on the time required for the determination process by the determination means and the calculation process by the calculation means. [Item 5] 5. An inference device according to any one of items 2 to 4, characterized in that the auxiliary data input for each intermediate layer is changed. [Item 6] The inference device described in any one of items 1 to 5, characterized in that the control means sets auxiliary data of input data judged to be appropriate by the judgment means to a predetermined fixed value, and sets auxiliary data of input data judged to be inappropriate by the judgment means to a predetermined alternative data. [Item 7] the determining means calculates evaluation information for each of the plurality of types of input data; 7. The inference device according to item 6, characterized in that the suitability of the multiple types of input data for use in the inference process is determined by comparing the evaluation information with a predetermined threshold value. [Item 8] The inference device described in item 6 or 7, characterized in that when the judgment means judges that all of the multiple types of input data are appropriate, the control means sets the auxiliary data of the input data judged to be appropriate to the predetermined fixed value. [Item 9] The inference device described in item 8 is characterized in that the predetermined fixed value is 0 or NULL, and the predetermined alternative data is data determined based on characteristic information of input data determined to be appropriate by the determination means. [Item 10] The inference device described in item 7 is characterized in that, when the judgment means judges all of the multiple types of input data to be inappropriate, the control means appropriately changes the judgment result of the input data with the highest evaluation information. [Item 11] the calculation means calculates a plurality of pieces of characteristic information for each of the plurality of types of input data; 11. An inference device according to any one of items 1 to 10, wherein the control means determines the auxiliary data based on a plurality of pieces of characteristic information for each of the input data. [Item 12] Item 12. The inference device according to item 11, wherein the characteristic information is a normalized value of the characteristic acquired by the calculation means for each of the input data. [Item 13] 13. The inference device according to any one of items 1 to 12, wherein the plurality of types of input data include image data and sound data. [Item 14] When the input data is image data, the characteristic information includes a value related to any one of the amount of blur, brightness, or noise corresponding to sensitivity at the time of capturing the image data, Item 14. An inference device according to item 13, characterized in that, when the input data is sound data, it includes a value relating to volume or noise sound. [Item 15] When the input data is the image data, the characteristic information includes first characteristic information relating to a noise amount according to a sensitivity at the time of capturing the image data, and second characteristic information relating to a blur amount of the image data, When the input data is sound data, the input data includes first characteristic information related to a noise sound and second characteristic information related to a volume; Item 15. The inference device according to item 14, wherein the control means determines the auxiliary data based on the first characteristic information and the second characteristic information for each of the input data. [Item 16] An inference device described in any of items 2 to 5, characterized in that the inference model is subjected to a learning process in advance using learning data including input data for learning and teacher data. [Item 17] Item 17. An inference device according to item 16, characterized in that the learning input data includes a dataset that combines multiple types of input data that the judgment means has judged to be appropriate, or a dataset that combines input data that the judgment means has judged to be appropriate with auxiliary data that is determined based on characteristic information of the input data. [Item 18] The inference model is a neural network, An inference device described in any one of items 2, 3, 4, 5, 16, and 17, characterized in that the inference processing is deep learning using a neural network. [Item 19] 1. A control method for an inference device that executes inference processing using multiple types of input data and auxiliary data for each of the input data, comprising: a step of determining suitability of a plurality of types of input data input to the inference device for use in the inference process; calculating characteristic information for each of the plurality of types of input data; determining auxiliary data for the input data based on the suitability of the input data and characteristic information; A control method characterized in that, in the inference process, input data that is determined to be inappropriate by the judgment among the multiple types of input data and auxiliary data for each of the input data is replaced with auxiliary data determined based on characteristic information of the input data that is determined to be appropriate by the judgment. [Item 20] A program for causing a computer to function as an inference device described in any one of items 1 to 18. [Explanation of symbols]

[0101] 100... inference device, 101... control unit, 102... inference unit, 103... aptitude determination unit, 104... characteristic information calculation unit

Claims

1. 1. An inference device, comprising: an inference means for executing an inference process using a plurality of types of input data and auxiliary data for each of the input data; a determination means for determining suitability of a plurality of types of input data input to the inference device for use in the inference process; a calculation means for calculating characteristic information for each of the plurality of types of input data; a control means for determining auxiliary data for the input data based on the suitability of the input data and characteristic information; The inference device is characterized in that the inference means replaces input data that is determined to be inappropriate by the judgment means, among the multiple types of input data and auxiliary data for each of the input data, with auxiliary data determined based on characteristic information of the input data that is determined to be appropriate by the judgment means.

2. the inference means executes an inference process by applying the plurality of types of input data and auxiliary data for each of the input data to an inference model; The inference device according to claim 1, characterized in that the inference model includes an input layer to which the multiple types of input data are input, and an intermediate layer to which auxiliary data for each of the input data is input.

3. The inference device according to claim 2, characterized in that the intermediate layer outputs input data judged to be appropriate by the judgment means to a subsequent stage, and outputs auxiliary data determined based on characteristic information of the input data judged to be appropriate by the judgment means to a subsequent stage in place of input data judged to be inappropriate by the judgment means.

4. 3. The inference device according to claim 2, wherein the number of intermediate layers is changed depending on the time required for the determination process by said determination means and the calculation process by said calculation means.

5. 3. The inference device according to claim 2, wherein auxiliary data to be input is changed for each intermediate layer.

6. The inference device according to claim 1, characterized in that the control means sets auxiliary data of input data judged to be appropriate by the judgment means to a predetermined fixed value, and sets auxiliary data of input data judged to be inappropriate by the judgment means to a predetermined alternative data.

7. the determining means calculates evaluation information for each of the plurality of types of input data; 7. The inference device according to claim 6, wherein the suitability of the plurality of types of input data for use in the inference process is determined by comparing the evaluation information with a predetermined threshold value.

8. The inference device according to claim 6, characterized in that, when the judgment means judges that all of the multiple types of input data are appropriate, the control means sets the auxiliary data of the input data judged to be appropriate to the predetermined fixed value.

9. 9. The inference device according to claim 8, wherein the predetermined fixed value is 0 or NULL, and the predetermined alternative data is data determined based on characteristic information of input data determined to be appropriate by the determination means.

10. The inference device according to claim 7, characterized in that the control means appropriately changes the judgment result of the input data with the highest evaluation information when the judgment means judges all of the multiple types of input data to be inappropriate.

11. the calculation means calculates a plurality of pieces of characteristic information for each of the plurality of types of input data; 2. The inference device according to claim 1, wherein said control means determines said auxiliary data based on a plurality of pieces of characteristic information for each of said input data.

12. 12. The inference device according to claim 11, wherein the characteristic information is a normalized value of the characteristic obtained by the calculation means for each of the input data.

13. 2. The inference device according to claim 1, wherein the plurality of types of input data include image data and sound data.

14. When the input data is image data, the characteristic information includes a value related to any one of the amount of blur, brightness, or noise corresponding to sensitivity at the time of capturing the image data, 14. The inference device according to claim 13, wherein the input data, when said input data is sound data, includes a value relating to volume or noise sound.

15. When the input data is the image data, the characteristic information includes first characteristic information relating to a noise amount according to a sensitivity at the time of capturing the image data, and second characteristic information relating to a blur amount of the image data, When the input data is sound data, the input data includes first characteristic information related to a noise sound and second characteristic information related to a volume, 15. The inference device according to claim 14, wherein said control means determines said auxiliary data based on said first characteristic information and said second characteristic information for each of said input data.

16. The inference device according to claim 2, wherein the inference model is pre-trained using training data including input data for learning and teacher data.

17. The inference device described in claim 16, characterized in that the learning input data includes a dataset that combines multiple types of input data that the judgment means has determined to be appropriate, or a dataset that combines input data that the judgment means has determined to be appropriate with auxiliary data that is determined based on characteristic information of the input data.

18. The inference model is a neural network, 3. The inference device according to claim 2, wherein the inference processing is deep learning using a neural network.

19. 1. A control method for an inference device that executes inference processing using multiple types of input data and auxiliary data for each of the input data, comprising: a step of determining suitability of a plurality of types of input data input to the inference device for use in the inference process; calculating characteristic information for each of the plurality of types of input data; determining auxiliary data for the input data based on the suitability of the input data and characteristic information; A control method characterized in that, in the inference process, input data that is determined to be inappropriate by the judgment among the multiple types of input data and auxiliary data for each of the input data is replaced with auxiliary data determined based on characteristic information of the input data that is determined to be appropriate by the judgment.

20. A program for causing a computer to function as the inference device according to any one of claims 1 to 18.

Images