Inference method, inference device, and program

Abstract

To perform inference processing robust against unknown degradation occurring in input data.SOLUTION: An inference device comprises: a conversion parameter designation unit which designates one or more conversion parameters; a conversion unit which performs conversion based on the respective designated conversion parameters, on input data to generate converted data; a confidence map generation unit which generates a confidence map being data showing respective features of the converted data with respect to each piece of the converted data; an inverse conversion unit which performs inverse conversion with respect to the conversion performed when generating the converted data corresponding to the confidence maps respectively, on the respective confidence maps on the basis of the respective designated conversion parameters to generate inversely converted data; an integration unit which performs integration processing of integrating respective pieces of the inversely converted data to generate integrated data of which the number of dimensions is consistent with the confidence maps of the input data; and an analysis unit which performs analysis processing on the integrated data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an inference method, an inference device, and a program. [Background technology]

[0002] Deep convolutional neural networks (DCNNs), which are formed by stacking convolutional neural networks (CNNs), have become a mainstream technique in computer vision and image processing in recent years. DCNNs have significantly contributed to improving the performance of computer vision tasks such as image recognition, object detection, and semantic segmentation. In particular, semantic segmentation has become a key element in many vision applications, such as video surveillance, medical image processing, and autonomous driving.

[0003] Existing semantic segmentation algorithms exhibit high recognition accuracy when targeting clean images, i.e., images without image degradation. In contrast, images actually obtained from applications such as video surveillance and autonomous driving often suffer from common image degradation such as noise, blurring, and compression artifacts. However, even common image degradation is image degradation that semantic segmentation algorithms do not anticipate. When such image degradation occurs in a target image, a domain shift occurs in the distribution of training data, as shown in, for example, Non-Patent Document 1. Therefore, when existing semantic segmentation algorithms are applied directly to images with image degradation, recognition accuracy is significantly reduced. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Christoph Kamann and Carsten Rother, “Benchmarking the Robustness of Semantic Segmentation Models”, International Journal of Computer Vision, 2021. [Non-Patent Document 2] Kazuki Endo, Masayuki Tanaka, and Masatoshi Okutomi, “CLASSIFYING DEGRADED IMAGES OVER VARIOUS LEVELS OF DEGRADATION”, in 2020 IEEE International Conference on Image Processing (ICIP), 2020. [Non-Patent Document 3] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising”, IEEE transactions on image processing, 2017. [Non-Patent Document 4] Dazhou Guo, Yanting Pei, Kang Zheng, Hongkai Yu, Yuhang Lu, and SongWang, “Degraded Image Semantic Segmentation With Dense-Gram Networks”, IEEE Transactions on Image Processing, 2019. [Non-Patent Document 5] Christoph Kamann and Carsten Rother, “Increasing the Robustness of Semantic Segmentation Models with Painting-by-Numbers”, in European Conference on Computer Vision, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] A naive approach to dealing with image degradation is to combine image restoration and semantic segmentation, as shown in Fig. 12. However, many existing image restoration algorithms are algorithms constructed to correspond to specific degradation models such as Gaussian noise, compression distortion, and blur (see, for example, Non-Patent Documents 2 and 3). In other words, existing image restoration algorithms are algorithms constructed assuming a degradation model in advance, and are effective for images containing image degradation of the assumed degradation model, but cannot restore images containing image degradation other than the assumed degradation model.

[0006] For example, as shown in FIG. 12, assume that image data 300-1 includes expected image degradation and image data 300-2 includes unexpected image degradation. Applying an image restoration algorithm corresponding to the expected image degradation occurring in image data 300-1 to these image data 300-1 and 300-2 results in restored image data 310-1 and 310-2 corresponding to the image data 300-1 and 300-2, respectively. The restored image data 310-1 is properly restored. In contrast, the restored image data 310-2 is not properly restored. Therefore, even if semantic segmentation is performed on each of the restored image data 310-1 and 310-2, the following results are obtained: In other words, the recognition result image data 320-1 obtained from the restored image data 310-1 correctly identifies the class for each pixel. In contrast, the recognition result image data 320-2 obtained from the restored image data 310-2 incorrectly identifies the class for each pixel.

[0007] In contrast to the above-mentioned naive approach, Non-Patent Document 4 proposes a new neural network that performs semantic segmentation on images that include image degradation. However, the neural network proposed in Non-Patent Document 4 needs to be trained for each degradation model. Therefore, like the above-mentioned naive approach, the technology disclosed in Non-Patent Document 4 cannot correctly perform semantic segmentation on images that have been degraded by an unknown degradation model.

[0008] In practical situations, the degradation model is often unknown. Non-Patent Document 5 proposes a learning scheme that is robust against various degradations. However, the technique disclosed in Non-Patent Document 5 has a problem in that it needs to be re-learned at least once in order to be effective against typical degradations such as blurring.

[0009] The present invention has been made to solve the above problem, and its purpose is to provide an inference method, an inference device, and a program that perform inference processing that is robust against unknown degradation occurring in input data. [Means for solving the problem]

[0010] In order to solve the above problem, one aspect of the present invention is an inference method including: a transformation parameter specification step of specifying one or more transformation parameters; a transformation step of performing a transformation on input data based on each of the specified transformation parameters to generate transformed data; a reliability map generation step of generating, for each of the transformed data, a reliability map that is data indicating the characteristics of each of the transformed data; an inverse transformation step of performing, for each of the reliability maps, an inverse transformation of the transformation performed when the corresponding transformed data was generated based on each of the specified transformation parameters to generate inverse transformed data; an integration step of performing an integration process on each of the inverse transformed data to generate integrated data whose number of dimensions matches the reliability map of the input data; and an analysis step of performing an analytical process on the integrated data.

[0011] One aspect of the present invention is an inference device in the invention described above, comprising: a transformation parameter designation unit that designates one or more transformation parameters; a transformation unit that performs transformation on input data based on each of the designated transformation parameters to generate transformed data; a reliability map generation unit that generates, for each of the transformed data, a reliability map that is data indicating the characteristics of each of the transformed data; an inverse transformation unit that performs, for each of the reliability maps, an inverse transformation of the transformation that was performed when the corresponding transformed data was generated based on each of the designated transformation parameters to generate inversely transformed data; an integration unit that performs an integration process to integrate each of the inversely transformed data and generates integrated data whose number of dimensions matches the reliability map of the input data; and an analysis unit that performs an analytical process on the integrated data.

[0012] One aspect of the present invention is a program for causing a computer to execute the following steps in the invention described above: specifying one or more transformation parameters; performing a transformation on input data based on each of the specified transformation parameters to generate transformed data; generating, for each of the transformed data, a reliability map that is data indicating characteristics of each of the transformed data; performing, for each of the reliability maps, an inverse transformation of the transformation that was performed when the corresponding transformed data was generated, based on each of the specified transformation parameters to generate inverse-transformed data; performing an integration process to integrate each of the inverse-transformed data to generate integrated data whose number of dimensions matches that of the reliability map of the input data; and performing an analytical process on the integrated data. [Effects of the Invention]

[0013] The present invention enables inference processing that is robust against unknown degradations occurring in input data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing the configuration of an inference device according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an overview of a technique adopted in the inference device of the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating a flow of processing performed by the inference device of the first embodiment. [Figure 4] 1 is a block diagram showing a configuration of a condition selection device according to a first embodiment. [Figure 5] FIG. 3 is a diagram showing a flow of processing performed by the condition selection device of the first embodiment. [Figure 6] 4 is a diagram illustrating a data format of a table generated in a processing result storage unit included in the condition selection device of the first embodiment. FIG. [Figure 7] FIG. 10 is a block diagram showing the configuration of an inference device according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating a flow of processing performed by the inference device of the second embodiment. [Figure 9] FIG. 10 is a block diagram showing the configuration of a conversion parameter selection device according to a third embodiment. [Figure 10] FIG. 10 is a diagram showing the flow of processing performed by a conversion parameter selection device according to a third embodiment. [Figure 11] FIG. 11 is a diagram illustrating a data format of a table generated in a processing result storage unit included in the conversion parameter selection device of the third embodiment. [Figure 12] FIG. 1 shows an overview of a naive approach for applying semantic segmentation to image data with image degradation. DETAILED DESCRIPTION OF THE INVENTION

[0015] (First embodiment) Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an inference device 1 according to a first embodiment. FIG. 2 is a diagram showing an overview of a technique employed by the inference device 1 according to the first embodiment shown in FIG. 1. Image data 100-1 shown in FIG. 2 is image data to be recognized by semantic segmentation, obtained from applications such as video surveillance and autonomous driving. Here, it is assumed that unknown image degradation occurs in the image data 100-1. The technique employed by the inference device 1 is based on the following two observations: (1) Low-resolution image data 100-2, ..., 100-N obtained by, for example, performing image conversion (downscaling) on ​​the image data 100-1 loses patterns such as fine textures, but also reduces image degradation such as JPEG (Joint Photographic Experts Group) compression and blurring. Therefore, the low-resolution image data 100-2, ..., 100-N become image data with reduced image degradation. (2) The semantic segmentation algorithm is also effective for low-resolution image data 100-2, ..., 100-N. In other words, there is not much difference in the accuracy of semantic segmentation between low-resolution image data without image degradation and high-resolution image data without image degradation.

[0016] Based on the above two observations (1) and (2), the method employed in the inference device 1 performs semantic segmentation on image data 100-1, including image degradation, as follows: An arbitrarily defined image transformation is performed on the image data 100-1 to obtain image data 100-2, ..., 100-N. For example, an encoder-decoder type semantic segmentation neural network, such as SegNet or U-Net, is used to separate the encoder and decoder portions of a trained neural network. Downsampling is performed by providing each of the image data 100-1, 100-2, ..., 100-N to the neural network of the encoder portion, thereby obtaining corresponding confidence maps 110-1, 110-2, ..., 110-N. Here, the confidence map is data known as a feature map, which indicates the features contained in the image data 100-1, 100-2, ..., 100-N. The value of each element of the reliability map, i.e., the value indicating the feature, is called a logit. Based on the logit value, the most reliable class for each pixel of the original image data is assigned as the class corresponding to that pixel. Here, the most reliable class for each pixel is, for example, the class with the largest logit value corresponding to each pixel, or the class with the largest normalized logit value. The class refers to the type of object, such as "person" or "car," contained in each of the image data 100-1, 100-2, ..., 100-N.

[0017] For each of the reliability maps 110-2, ..., 110-N, an inverse transformation corresponding to the image transformation performed when the corresponding image data 100-2, ..., 100-N was obtained is performed. The reliability map 110-1 and the inversely transformed reliability maps 110-2, ..., 110-N are integrated to generate integrated data. This method of integrating the reliability maps 110-1, ..., 110-N corresponds to a technique known as ensemble in machine learning. The generated integrated data is provided to the neural network of the decoder section for upsampling, thereby obtaining image data 150, which is the result of semantic segmentation. Note that the separated neural network of the encoder section and the neural network of the decoder section are used with the coefficients applied to the neurons, i.e., the weights and bias values, fixed in a learned state. Hereinafter, the state in which the coefficients are fixed in a learned state is also referred to as "coefficient freeze."

[0018] Before explaining each functional unit of the inference device 1, we will explain the meaning of variables and functions used in the explanation of each functional unit. Image data obtained from applications such as video surveillance and autonomous driving, which is input to the inference device 1 (hereinafter referred to as "input data"), is represented by the symbols in the following formula (1). Hereinafter, when referring to the symbols in the following formula (1), they will be written as vector x.

[0019]

number

[0020] For example, if the input data is RGB color image data, there will be three channel directions indicating the pixel values ​​of red, green, and blue for each pixel. In this case, the vector x will be three-dimensional data with vertical, horizontal, and channel directions. The calculation by the backbone neural network that calculates the confidence map from the vector x is expressed by the function S(·) shown in the following equation (2).

[0021]

number

[0022] The neural network that calculates the function S(·) is a trained neural network, and is used with its coefficients frozen. Specific examples of neural networks that calculate the function S(·) include the neural network in the encoder part of the encoder-decoder type neural network for semantic segmentation, such as the SegNet and U-Net mentioned above, and FCN (Fully Convolutional Network).

[0023] The reliability map obtained by applying the function S(·) to the vector x is expressed as the first symbol from the left, vector p, or the second symbol from the left, p, in the following equation (3). chw It is expressed as: p chw The subscript c is an index in the channel direction, h is an index in the vertical direction, and w is an index in the horizontal direction. In other words, the vector p representing the confidence map is three-dimensional data. Therefore, by determining a set of values ​​for c, h, and w, one feature at the position of c, h, and w in vector p is identified.

[0024]

number

[0025] The calculation for assigning a class to each pixel of vector x from vector p, which is a reliability map, is expressed by the function g(·) shown in the following equation (4).

[0026]

number

[0027] The neural network that calculates the function g(·) is a trained neural network, similar to the neural network for the function S(·), and is used with its coefficients frozen. Specific examples of neural networks that calculate the function g(·) include the neural network for the decoder part of the encoder-decoder type neural network for semantic segmentation, such as the SegNet and U-Net mentioned above, and a neural network that performs upsampling, restoring the size of the vector p to the size of the original input data, vector x.

[0028] The result of semantic segmentation, i.e., the recognition result data obtained by applying the function S(·) to the vector x and then applying the function g(·) to the result of the function S(·), is represented by the symbol on the left side of the following equation (5). Hereinafter, in this text, the circumflex vector y on the left side of the following equation (5) will be referred to as "vector ^y".

[0029]

number

[0030] (Configuration of the inference device of the first embodiment) As shown in Figure 1, the inference device 1 includes a conversion integration condition storage unit 10, a conversion parameter designation unit 11, a data acquisition unit 12, a conversion unit 13, a reliability map generation unit 14, an inverse conversion unit 15, an integration unit 16, an analysis unit 17, and an output unit 18. The conversion integration condition storage unit 10 stores N pre-selected conversion parameters and one pre-selected integration arithmetic expression, where N is an integer equal to or greater than 2. The set of conversion parameters stored in the conversion integration condition storage unit 10 is expressed by the following equation (6).

[0031]

number

[0032] ξ1,…,ξ in equation (6) n ,…,ξN Each of these symbols indicates an individual transformation parameter, and "{·}" is a symbol that indicates a set. Here, n is an arbitrary integer between 1 and N. Hereinafter, the transformation parameter ξ n When the expression "ξ1" is used, it indicates any one of the transformation parameters. The transformation parameter ξ1 is a transformation parameter that does not perform image transformation. As shown in FIG. 2, the image data 100-1, which is the original input data, is subjected to the function S(·) without undergoing image transformation, and the reliability map 110-1 obtained by applying the function S(·) is the target of integration. Therefore, the transformation parameters ξ1, ..., ξ n ,…,ξ N must contain one transformation parameter that does not perform transformation. Therefore, here, the transformation parameter ξ1 is set as the transformation parameter that does not perform image transformation.

[0033] Transformation parameters ξ2~ξ for N=2 or more N are the transformation parameters for image transformation. N are transformation parameters that indicate, for example, geometric image transformations such as linear transformations such as enlargement, rotation, affine transformation, and projective transformation, as well as nonlinear transformations using B-spline interpolation, in addition to the image transformation of reduction described above. N The transformation parameters ξ2 to ξ may be transformation parameters that indicate optical image transformations such as transformations of color tone or color temperature. N may be parameters indicating image transformation related to improving image quality (hereinafter referred to as "image quality") such as removing or adding noise, removing or adding blur, or deteriorating image quality. Here, noise is not limited to general white noise, but may also be compression noise, hostile noise that may cause erroneous recognition, etc. Note that the transformation parameters ξ2 to ξ N Each of the parameters ξ includes data indicating the type of image transformation, as well as a variable indicating the ratio of the transformation, for example, in the case of a reduction image transformation. nIf ξ is a transformation parameter that indicates a two-dimensional rotational image transformation, then one variable is included. n If is a transformation parameter indicating an affine transformation, six variables will be included.

[0034] The reason why transformation parameters indicating image transformations that add image degradation may be included in addition to transformation parameters indicating image transformations that reduce the effects of image degradation, such as image reduction transformations, is that it is assumed that existing semantic segmentation or neural networks for semantic segmentation that will be proposed in the future are robust against image transformations that increase image degradation. In this case, it is expected that adding image transformations that increase image degradation will ultimately result in more accurate recognition results from semantic segmentation.

[0035] The conversion parameter designation unit 11 designates conversion parameters to be used by the conversion unit 13 and the inverse conversion unit 15. More specifically, the conversion parameter designation unit 11 designates N conversion parameters ξ1 to ξ2 stored in the conversion integration condition storage unit 10. N and the transformation parameters ξ1 to ξ N The conversion parameters are specified by outputting them one by one to the conversion unit 13 and the inverse conversion unit 15. The data acquisition unit 12 acquires input data, which is image data to be recognized, provided from the outside, and outputs the acquired input data to the conversion unit 13.

[0036] The conversion unit 13 converts the N conversion parameters ξ1 to ξ2 output by the conversion parameter designation unit 11 into the input data output by the data acquisition unit 12. N Based on each of the above, the image transformation expressed by the following equation (7) is performed.

[0037]

number

[0038] In equation (7), the right-hand side of D(·;ξ n ) is a function that converts the image data given as an argument into the transformation parameters ξ n The vector x with a tilde on the left side of equation (7) is a transformation function that performs image transformation calculations based on n (In the following text, we use the vector ~x n ) is the transformation parameter ξ n Here, we will use the vector x to represent the image transformation performed on the vector ~x. n The set of N pieces of converted data generated by the conversion unit 13 through image conversion is expressed by the following equation (8).

[0039]

number

[0040] As described above, the transformation parameter ξ1 is a transformation parameter that does not perform image transformation, and therefore, in equation (8), the vector ~x1 and the vector x1 are the same data. The reliability map generation unit 14 applies the function S(·) shown in equation (2) to each of the N pieces of transformed data included in the set of transformed data in equation (8), and generates N pieces of reliability maps, the vectors ~p1, ..., ~p n ,…~p N Generate.

[0041]

number

[0042] The inverse transformation unit 15 calculates the N transformation parameters ξ1 to ξ2 output by the transformation parameter designation unit 11. N For each of the N reliability maps generated by the reliability map generating unit 14, an inverse transformation corresponding to the image transformation performed when the corresponding transformation data was generated is performed based on each of the n-th transformation parameters ξ. nWhen the image transformation by is an image transformation of rotation by θ degrees, the inverse transformation unit 15 performs an image transformation of rotation by -θ degrees on the n-th reliability map, thereby performing an inverse transformation. The inverse transformation performed by the inverse transformation unit 15 is expressed by the following equation (10).

[0043]

number

[0044] In equation (10), U(·;ξ n ) is the transformation parameter ξ for the confidence map given as an argument. n The vector p on the left side of equation (10) is a transformation function that performs the inverse transformation corresponding to the image transformation based on n is that the inverse transformation unit 15 calculates the transformation parameter ξ n Based on the n-th confidence map, vector ~p n The data is obtained by inversely transforming the vector p n are referred to as inverse transformed data. Note that when the integration unit 16 described below integrates N pieces of inverse transformed data, it is necessary for the integration unit 16 to be able to compare feature quantities at spatially identical positions in each of the N pieces of inverse transformed data. Here, the "spatially identical positions" refers to the following positions: The N pieces of inverse transformed data are data generated based on the vector x1. Therefore, for example, a position in the vector x1 where (c, h, w) corresponds to the position (1, 1, 1) exists in each of the N pieces of inverse transformed data. In other words, the position of each of the N pieces of inverse transformed data corresponds to one point in the vector x1, and the positions of the N pieces of inverse transformed data that have the same corresponding positions in the vector x1 are spatially identical positions.

[0045] Therefore, in order to enable the integration unit 16 to compare feature quantities at the same spatial position in each of the N pieces of inverse transformed data, the inverse transform unit 15 performs a position alignment process so that positions indicated by the same c, h, and w in the N pieces of inverse transformed data become the same spatial position, thereby generating N pieces of inverse transformed data. The set of N pieces of inverse transformed data generated by the inverse transform unit 15 is expressed by the following equation (11).

[0046]

number

[0047] The integration unit 16 integrates N pieces of inverse transformed data included in the set of inverse transformed data of equation (11) based on one pre-selected integration calculation formula stored in the transformation integration condition storage unit 10. The integration process performed by the integration unit 16 is expressed by the following equation (12) using a concatenation operator ([·]).

[0048]

number

[0049] The left side of equation (12) is a symbol indicating integrated data generated by the integration unit 16 by integrating N pieces of inverse transformed data, and hereinafter, the integrated data will be referred to as vector ^p. The integration unit 16 performs the integration process so that the number of dimensions of the integrated data generated by the integration process, i.e., vector ^p, is the same as the number of dimensions of the reliability map obtained by applying the function S(·) shown in equation (2) to the input data vector x, i.e., vector p in equation (3). Here, the same number of dimensions means that the data sizes represented by c×h×w are the same. More specifically, if the vector p in equation (3) is, for example, 8×256×256 in size, the size of the integrated data vector ^p is also 8×256×256.

[0050] The concatenation operator in equation (12) may simply be an integration calculation formula that performs an integration process of concatenating N feature amounts at positions where c, h, and w are the same in each of the N pieces of inverse transformed data, that is, the sum of the N feature amounts is used as the feature amount of the integrated data at that position. The concatenation operator in equation (12) may be any of the following integration calculation formulas:

[0051] For example, the concatenation operator in equation (12) may be five integration equations expressed by the following equations (13) to (17): The integration equation expressed by the following equation (13) is an integration equation that performs integration processing so that, in each of the N pieces of inverse transformed data, the maximum feature value among the N feature values ​​at a position where c, h, and w are the same is set as the feature value of the integrated data at that position.

[0052]

number

[0053] The integration formula expressed by the following formula (14) is an integration formula that performs integration processing so that the average value of N feature quantities at positions where c, h, and w are the same in each of N pieces of inverse transformed data is used as the feature quantity of the integrated data at that position.

[0054]

number

[0055] The integrated calculation formula expressed by the following formula (15) is a weight w n This is an integration formula that performs integration processing so that the largest multiplication value among the multiplication values ​​obtained by multiplying the above values ​​is used as the feature of the integrated data at that position. This largest multiplication value is also called the weighted maximum value.

[0056]

number

[0057] The integrated calculation formula expressed by the following formula (16) is a weight w that is predetermined for each of the N pieces of inverse transformation data for the N feature amounts at positions where c, h, and w are the same in each of the N pieces of inverse transformation data. n The multiplied values ​​are used to calculate the N weights w n The integration calculation formula is used to perform integration processing so that the divided value obtained by dividing the value of the sum of the values ​​of (a) and (b) is used as the feature of the integrated data at that position. The divided value is also called a weighted average value or a weighted mean value.

[0058]

number

[0059] The integration formula expressed by the following formula (17) is an integration formula that performs integration processing so that when N feature amounts at positions where c, h, and w are the same in each of N pieces of inverse transformed data are sorted in descending order, the average value of the top k feature amounts is used as the feature amount of the integrated data at that position.

[0060]

number

[0061] More specifically, the integrated calculation formula of formula (17) is expressed by the following formula (18).

[0062]

number

[0063] φ in Equation (18) (k) nchw is defined by the following equation (19).

[0064]

number

[0065] In equation (19), the function R(·) is a function that outputs the rank of the value given as an argument as a return value. For example, the vectors p1 to p N In the nth vector p n p nchw is applied as an argument to the function R(·). In this case, the function R(·) is nchw and N p 1chw ~p Nchw When the features of the argument p are sorted in descending order, nchw The rank of the input data is output as a return value. The value of k is an integer between 1 and N, and is a predetermined integer value. When k=1, formula (17) is the same as formula (13), that is, the integration calculation formula that performs integration processing to select the maximum value, and when k=N, formula (17) is the same as formula (14), that is, the integration calculation formula that performs integration processing to select the average value.

[0066] The concatenation operator in equation (12) may be an integration calculation formula as follows. For example, it may be an integration calculation formula that performs integration processing so that the median feature value among N feature values ​​at a position where c, h, and w are the same in each of N pieces of inverse transformed data is used as the feature value of the integrated data at that position. For each of N pieces of inverse transformed data, a weight w whose value is predetermined for each of the N pieces of inverse transformed data is used for the N feature values ​​at a position where c, h, and w are the same. n The integration formula may be an integration formula that performs integration processing so that the product of the median of the multiplied values ​​obtained by multiplying the values ​​of the above and the above is used as the feature of the integrated data at that position. The multiplied value is also called a weighted median.

[0067] The concatenation operator in equation (12) may be the following integration formula. If a certain position in the integrated data is c1, h1, w1, when calculating the feature quantities at the position of c1, h1, w1 using the integration formula, integration processing may be performed by applying the integration formula shown in equations (13) to (16) above, not only targeting the feature quantities at the positions of c1, h1, w1 in each of the N pieces of inverse transformed data, but also including feature quantities at positions near the positions of c1, h1, w1. In this case, the type of transformation parameter that takes neighborhood into consideration may be determined in advance, and the inverse transformed data corresponding to the predetermined type of transformation parameter may include the feature quantities of the neighborhood, while the inverse transformed data corresponding to types other than the predetermined type of transformation parameter may not include the feature quantities of the neighborhood. When applying the integration formula shown in equation (15) or (16), the weight w may be calculated for the inverse transformed data corresponding to the predetermined type of transformation parameter after including the feature quantities of the neighborhood. n For inverse transformation data corresponding to types other than the predetermined transformation parameter types, the weight w n The value of may be reduced.

[0068] Here, "nearby positions" may be, for example, 26 positions adjacent to the positions of c1, h1, and w1 in the up, down, left, right, front, back, and diagonal directions relative to the positions of c1, h1, and w1 when the positions of c1, h1, and w1 are used as a reference, or 6 positions adjacent to the positions of c1, h1, and w1 in the up, down, left, right, front, back, and diagonal directions relative to the positions of c1, h1, and w1, or positions included in any range that includes the positions of c1, h1, and w1 and is included in a predetermined range.

[0069] The integrated arithmetic formula may be a combination of multiple integrated arithmetic formulas. For example, the integrating unit 16 may integrate the integrated data generated based on equation (13) and the integrated data generated based on equation (14) further based on equation (13), and the result may be the final integrated data.

[0070] The analysis unit 17 performs a process of applying the function g(·) of equation (4) to the vector ^p, which is integrated data generated by the integration unit 16 by integrating N pieces of inverse transformed data. The process of applying the function g(·) of equation (4) is a recognition process, and more specifically, an image segmentation process that divides image regions into classes, which is performed in a semantic segmentation algorithm. The recognition process by the analysis unit 17 is expressed by the following equation (20).

[0071]

number

[0072] As shown in equation (20), the recognition process by the analysis unit 17 generates the vector ^y, which is recognition result data, i.e., data indicating the result of semantic segmentation. The output unit 18 may be, for example, a display device equipped with a screen such as a liquid crystal display, or a storage device such as a semiconductor memory or an HDD (Hard Disk Drive). When the output unit 18 is a display device, the analysis unit 17 displays the vector ^y, which is the recognition result data generated by the recognition process, on the screen. When the output unit 18 is a storage device, the analysis unit 17 writes the vector ^y, which is the recognition result data generated by the recognition process, to the output unit 18 for storage.

[0073] (Processing by the inference device of the first embodiment) 3 is a flowchart showing the flow of processing by the inference device 1. When the inference device 1 is started, the conversion parameter designation unit 11 designates N conversion parameters ξ1 to ξ2 stored in advance in the conversion integration condition storage unit 10. N and the transformation parameters ξ1 to ξ Nand writes and stores it in an internal storage area (step Sa1). The integration unit 16 reads out one integration arithmetic expression stored in advance in the conversion integration condition storage unit 10, and writes and stores the read integration arithmetic expression in an internal storage area (step Sa2). The data acquisition unit 12 acquires input data, which is image data of the recognition target provided from the outside, and outputs the acquired input data to the conversion unit 13 (step Sa3).

[0074] The conversion parameter designation unit 11 designates the conversion parameters ξ1 to ξ2 stored in an internal storage area. N One of the transformation parameters ξ n The transformation parameter designation unit 11 selects and reads out the read transformation parameter ξ n to the conversion unit 13 and the inverse conversion unit 15. This starts the first process of the loop La1s to La1e.

[0075] The conversion unit 13 converts the input data output by the data acquisition unit 12 and the conversion parameters ξ output by the conversion parameter designation unit 11 into n The conversion unit 13 converts the input data vector x into the conversion parameters ξ n The conversion unit 13 performs image conversion expressed by equation (7) based on the vector ∼x to generate converted data. n is output to the reliability map generating unit 14 (step Sa4).

[0076] The reliability map generating unit 14 converts the vector ∼x n and take in the resulting vector ~x n The reliability map generating unit 14 generates a reliability map by applying the function S(·) shown in Equation (2) to the vector ∼p n is output to the inverse conversion unit 15 (step Sa5).

[0077] The inverse transformation unit 15 uses the reliability map output by the reliability map generation unit 14 and the transformation parameters ξ output by the transformation parameter specification unit 11.n The inverse transformation unit 15 takes in the vector ∼p n For the imported transformation parameters ξ n Based on this, the inverse transformation expressed by the formula (10) is performed, that is, the transformation unit 13 calculates the transformation parameter ξ n The inverse transformation unit 15 generates inverse transformation data by performing an inverse transformation corresponding to the image transformation performed based on the vector p n to the integration unit 16 (step Sa6).

[0078] The conversion parameter designation unit 11 designates the conversion parameters ξ1 to ξ2 stored in an internal storage area. N Any one of the unselected transformation parameters ξ n , that is, any one of the transformation parameters ξ that has not been output to the transformation unit 13 and the inverse transformation unit 15 n This repeatedly selects the transformation parameters ξ1 to ξ N The processing of steps Sa4 to Sa6 corresponding to each of the above is repeatedly performed (loops La1s to La1e). n is a transformation parameter ξ1 indicating that image transformation is not performed, the transformation unit 13 and the inverse transformation unit 15 perform the following processes. In the process of step Sa4, the transformation unit 13 outputs the vector x, which is input data acquired from the data acquisition unit 12, as a vector ~x1 to the reliability map generation unit 14 without performing image transformation. In the process of step Sa6, the inverse transformation unit 15 outputs the vector ~p1, which is the reliability map output by the reliability map generation unit 14, to the integration unit 16 as a vector p1 without performing inverse transformation.

[0079] The integration unit 16 refers to the transformation integration condition storage unit 10 and calculates the transformation parameters ξ1 to ξ N The integration unit 16 detects "N", which is the number of vectors p n The integration unit 16 repeatedly takes in the taken vector pn While the number of elements does not match N, the vector p n Meanwhile, the integration unit 16 continues to take in the taken vector p n When the number of vectors is equal to N, the N vectors p are combined based on the integrated formula stored in the internal memory. n The integration unit 16 integrates the vectors p to generate integrated data. The integration unit 16 outputs the generated integrated data, that is, the vector ^p, to the analysis unit 17 (step Sa7).

[0080] The analysis unit 17 takes in the vector ^p, which is the integrated data output by the integration unit 16, and applies the function g(·) shown in equation (4) to the taken-in vector ^p to perform the recognition processing shown in equation (20) to generate recognition result data. The analysis unit 17 outputs the vector ^y, which is the recognition result data, to the output unit 18 (step Sa8). The output unit 18 takes in the vector ^y, which is the recognition result data output by the analysis unit 17. As described above, if the output unit 18 is a display device, the output unit 18 displays the taken-in recognition result data vector ^y on a screen. If the output unit 18 is a storage device, the output unit 18 stores the taken-in recognition result data vector ^y (step Sa9). This completes the inference processing performed by the inference device 1 for one piece of input data. When the next input data is provided to the data taking-in unit 12, the processing of steps Sa3 to Sa9 is performed again.

[0081] (Configuration of the condition selection device of the first embodiment) In order for the inference device 1 described above to obtain highly accurate recognition results, it is necessary to select a combination of a plurality of conversion parameters stored in advance in the conversion integration condition storage unit 10 and a single integration calculation formula so as to be an optimal combination for image degradation occurring in the input data. The condition selection device 2 shown in Fig. 4 is a device that selects this optimal combination.

[0082] The input data provided to the data acquisition unit 12 of the inference device 1 described above is assumed to be a plurality of image data arranged in time series and obtained from a specific application, such as an application for video surveillance or autonomous driving. The specific application acquires a plurality of image data captured at regular time intervals by a specific camera, for example. Because the acquired plurality of image data are captured by the same camera and acquired by the same application, each of the acquired plurality of image data will have common image degradation. Here, the image degradation occurring in each of the plurality of image data is unknown image degradation, but is common to each of the image data, and is assumed to be image degradation to the extent that a human can visually distinguish the class of each pixel of the image data.

[0083] Figure 4 is a block diagram showing the configuration of a condition selection device 2 according to the first embodiment. In the condition selection device 2 shown in Figure 4, the same components as those of the inference device 1 shown in Figure 1 are given the same reference numerals, and only the different components will be described below. The condition selection device 2 includes a conversion integration condition storage unit 10, a conversion parameter designation unit 11a, a conversion unit 13, a reliability map generation unit 14, an inverse conversion unit 15, an integration unit 16a, an analysis unit 17, a training data storage unit 21, a data reading unit 22, an integrated arithmetic expression selection unit 23, and a processing result storage unit 24.

[0084] The training data storage unit 21 stores multiple training data sets in advance. As described above, each of the multiple image data sets arranged in time series and obtained from a specific application contains unknown image degradation. However, the image degradation is such that a human can visually distinguish the class of each pixel in the image data. Therefore, instead of providing all of the multiple image data sets, i.e., multiple input data sets, obtained from the specific application to the inference device 1, a portion of the input data sets is selected as training data to be applied to the condition selection device 2. Correct answer data indicating the class of each pixel corresponding to each of the selected multiple training input data sets is generated. Each of the generated correct answer data sets is associated with the corresponding training input data set to generate multiple training data sets. In other words, one training data set includes one training input data set and one correct answer data set corresponding to the training input data set. The multiple training data sets generated in this manner are stored in advance in the training data storage unit 21. Here, the correct answer data set corresponding to one training input data set, vector x, is represented by the symbol shown in the following equation (21). In the following description, the symbol in equation (21) will be referred to as vector y.

[0085]

number

[0086] Each time the data reading unit 22 receives a start instruction signal or a continue instruction signal, it reads one piece of training data from the training data storage unit 21. If the data reading unit 22 is unable to read training data from the training data storage unit 21, it outputs an end instruction signal to the integrated arithmetic formula selection unit 23. If the data reading unit 22 is able to read training data from the training data storage unit 21, it outputs a start instruction signal to the conversion parameter designation unit 11a. The data reading unit 22 outputs the training input data included in the read piece of training data to the conversion unit 13, and outputs the correct answer data included in the training data to the integrated arithmetic formula selection unit 23.

[0087] The transformation parameter designation unit 11a designates N transformation parameters ξ1 to ξ2 that indicate image transformation given from the outside. N Upon receiving the start instruction signal, the conversion parameter designation unit 11a takes in the N conversion parameters ξ1 to ξ N One transformation parameter ξ n The conversion parameter designation unit 11a selects the selected conversion parameter ξ n to the transform unit 13 and the inverse transform unit 15, and the transform parameters ξ n Specify.

[0088] The integrating unit 16a receives M integrated arithmetic expressions provided from the outside and generates M indices for "integrated arithmetic expression 1," ..., "integrated arithmetic expression M," corresponding to each of the M integrated arithmetic expressions. Here, M is an integer greater than or equal to 2. The integrating unit 16a performs an integration process to integrate N pieces of inverse transformation data included in the set of inverse transformation data of Equation (11) based on each of the M integrated arithmetic expressions received, to generate integrated data. The integrating unit 16a outputs the indices corresponding to the integrated arithmetic expressions used in the integration process to the integrated arithmetic expression selection unit 23, one by one, in the order in which they were used. The integrating unit 16a outputs the integrated data to the analysis unit 17, one by one, in the order in which they were generated. In other words, when the integrating unit 16a performs an integration process based on a certain integrated arithmetic expression to generate integrated data, it outputs the index corresponding to the certain integrated arithmetic expression to the integrated arithmetic expression selection unit 23 and outputs the integrated data generated based on the certain integrated arithmetic expression to the analysis unit 17.

[0089] When M pieces of recognition result data are obtained for one piece of training data, the integrated arithmetic formula selection unit 23 outputs a continuation instruction signal to the data reading unit 22. Upon receiving the end instruction signal, the integrated arithmetic formula selection unit 23 selects one integrated arithmetic formula that is optimal for image degradation occurring in the input data included in the training data, based on all of the recognition result data obtained up until receiving the end instruction signal and the correct answer data corresponding to each of the recognition result data.

[0090] The processing result storage unit 24 stores, for each piece of supervised data output by the data reading unit 22 to the integrated arithmetic formula selection unit 23, the N conversion parameters output by the conversion parameter designation unit 11a to the integrated arithmetic formula selection unit 23 and the M pieces of recognition result data output by the analysis unit 17 to the integrated arithmetic formula selection unit 23 in association with each other. The conversion integration condition storage unit 10 included in the condition selection device 2 does not store any data in its initial state, and stores multiple conversion parameters and one integrated arithmetic formula when the selection process by the integrated arithmetic formula selection unit 23 is completed. The conversion integration condition storage unit 10 storing multiple conversion parameters and one integrated arithmetic formula will be used as the conversion integration condition storage unit 10 of the inference device 1 in FIG. 1.

[0091] (Processing by the condition selection device of the first embodiment) Fig. 5 is a flowchart showing the flow of processing by the condition selection device 2 of the first embodiment. Before the flowchart shown in Fig. 5 is started, a plurality of training data are written in advance to the training data storage unit 21 of the condition selection device 2. The conversion integration condition storage unit 10 and the processing result storage unit 24 are initialized to a state in which no data is stored.

[0092] The conversion parameter designation unit 11a of the condition selection device 2 selects N conversion parameters ξ1 to ξ2 given from the outside. N and take in the N transformation parameters ξ1 to ξ NThe integrating unit 16a writes and stores the M integrated arithmetic expressions into an internal storage area (step Sb1). The integrating unit 16a takes in M ​​integrated arithmetic expressions provided from the outside, and generates M indices of "integrated arithmetic expression 1", ..., "integrated arithmetic expression M" corresponding to each of the M integrated arithmetic expressions. The integrating unit 16a associates the M integrated arithmetic expressions with the M indices of "integrated arithmetic expression 1", ..., "integrated arithmetic expression M" corresponding to each of the M integrated arithmetic expressions, and writes and stores the M indices into an internal storage area. The integrating unit 16a outputs the generated indices of "integrated arithmetic expression 1", ..., "integrated arithmetic expression M" to the integrated arithmetic expression selecting unit 23. The integrated arithmetic expression selecting unit 23 takes in the indices of the M "integrated arithmetic expression 1", ..., "integrated arithmetic expression M". The integrated arithmetic formula selection unit 23 generates a table in the processing result storage unit 24 in the data format shown in FIG. 6, which has items for "correct data" and "conversion parameters" and items for M "integrated arithmetic formula 1", ..., "integrated arithmetic formula M" corresponding to each of the indexes of the M imported "integrated arithmetic formula 1", ..., "integrated arithmetic formula M" (step Sb2).

[0093] When the data reading unit 22 receives a start instruction signal from the outside (step Sb3), the first processing of loop Lb1s to Lb1e is started. The data reading unit 22 reads one piece of training data from the training data storage unit 21. After reading the training data, the data reading unit 22 outputs a start instruction signal to the conversion parameter designation unit 11a. The data reading unit 22 outputs the input data included in the read training data to the conversion unit 13. The conversion unit 13 takes in the input data output by the data reading unit 22. The data reading unit 22 outputs the correct answer data included in the read training data to the integrated arithmetic formula selection unit 23 (step Sb4).

[0094] The integrated arithmetic formula selection unit 23 takes in the correct data output by the data reading unit 22. Here, the vector y, which is the correct data taken in by the integrated arithmetic formula selection unit 23, is written as vector y1, vector y2, ... with a subscript indicating the order in which it was taken in. When the integrated arithmetic formula selection unit 23 takes in vector y1, which is the first correct data, it creates one record in the table of the processing result storage unit 24 and writes the taken in vector y1 in the "correct data" field of the created record.

[0095] When the conversion parameter designation unit 11a receives a start instruction signal from the data reading unit 22, it selects N conversion parameters ξ1 to ξ2 from an internal storage area. N and the N transformation parameters ξ1 to ξ N to the integrated arithmetic expression selection unit 23. The integrated arithmetic expression selection unit 23 selects the N transformation parameters ξ1 to ξ2 output by the transformation parameter designation unit 11a. N The integrated calculation formula selection unit 23 selects the N transformation parameters ξ1 to ξ N The transformation parameter specifying unit 11a writes the vector y1 into the "transformation parameter" field of the most recently generated record in the table of the processing result storage unit 24, that is, the record in which the vector y1 is written into the "correct label" field. N The conversion parameter specifying unit 11a outputs the value of "N", which is the number of conversion parameters, to the integrating unit 16a. The integrating unit 16a takes in the value of "N" output by the conversion parameter specifying unit 11a (step Sb5).

[0096] When the conversion parameter designation unit 11a receives a start instruction signal from the data reading unit 22, it reads N conversion parameters ξ1 to ξ2 stored in an internal storage area. N The transformation parameter ξ is one of n The transformation parameter designation unit 11a reads out the transformation parameter ξ nto the conversion unit 13 and the inverse conversion unit 15. This starts the initial processing of the loop Lb2s to Lb2e. The processing of the loop Lb2s to Lb2e is the same as the processing of the loop La1s to La1e shown in Fig. 3. Note that the processing of step Sb5 and the processing of the loop Lb2s to Lb2e are both started by the data reading unit 22 outputting the input data and the correct answer data in the processing of step Sb4, and therefore are performed in parallel.

[0097] The integration unit 16a refers to the value of "N" acquired in the process of step Sb5. During the process of loops Lb2s to Lb2e, the integration unit 16a integrates vectors p1 to p2, which are inverse transformation data output by the inverse transformation unit 15. N The integration unit 16a takes in the taken-in vector p n When the number of vectors p1 to p2 reaches N, the integration unit 16a selects one of the M integrated arithmetic expressions to which the indices "integrated arithmetic expression 1", ..., "integrated arithmetic expression M" are assigned and stored in the internal storage area, and the index of "integrated arithmetic expression m" associated with the integrated arithmetic expression. Here, m is an arbitrary integer between 1 and M. The integration unit 16a outputs the index of the selected "integrated arithmetic expression m" to the integrated arithmetic expression selection unit 23. The integration unit 16a selects the index of the selected "integrated arithmetic expression m" from the vectors p1 to p2 based on the selected integrated arithmetic expression. N The integrating unit 16a integrates the data to generate integrated data, and outputs the generated integrated data to the analyzing unit 17 (step Sb9).

[0098] The analysis unit 17 generates recognition result data by performing the same recognition processing as the processing of step Sa8 shown in FIG. 3. The analysis unit 17 outputs the generated recognition result data to the integrated arithmetic expression selection unit 23 (step Sb10). In the processing of step Sb9, the integrated arithmetic expression selection unit 23 takes in the index of "integrated arithmetic expression m" output by the integration unit 16a and the recognition result data output by the analysis unit 17 in the processing of step Sb10. The integrated arithmetic expression selection unit 23 detects the most recently generated record in the table of the processing result storage unit 24, i.e., the record in which the vector y1 is written in the "correct label" field. In the detected record, the integrated arithmetic expression selection unit 23 writes the captured recognition result data into the "integrated arithmetic expression m" field corresponding to the index of the captured "integrated arithmetic expression m." Here, when m=2, as shown in FIG. 6, the vector ^y 1,2 is written in the "integrated arithmetic formula 2" field (step Sb11).

[0099] The integration unit 16a selects any one of the unselected integrated arithmetic formulas. As a result, the processing of steps Sb9 to Sb11 is performed again (loop Lb3s to Lb3e). When the processing of steps Sb9 to Sb11 corresponding to each of the M integrated arithmetic formulas is completed, the processing of loop Lb3s to Lb3e ends. When the integrated arithmetic formula selection unit 23 writes the recognition result data into all items of "integrated arithmetic formula 1", ..., "integrated arithmetic formula M" of the record corresponding to vector y1 in the table of the processing result storage unit 24, it outputs a continuation instruction signal to the data reading unit 22.

[0100] When the data reading unit 22 receives a continue instruction signal from the integrated arithmetic formula selection unit 23, it reads out any one piece of training data that has not been read out as a processing target from the training data storage unit 21. After reading the training data, the data reading unit 22 outputs a start instruction signal to the conversion parameter designation unit 11a. As a result, the training data read out by the data reading unit 22 is subjected to processing in steps Sb4 and Sb5, loops Lb2s to Lb2e, and loops Lb3s to Lb3e (loops Lb1s to Lb1e). When all the training data stored in the training data storage unit 21 has been read out as a processing target, the data reading unit 22 is unable to read out any more training data and therefore does not output a start instruction signal to the conversion parameter designation unit 11a and outputs an end instruction signal to the integrated arithmetic formula selection unit 23.

[0101] When the integrated arithmetic formula selection unit 23 receives an end instruction signal from the data reading unit 22, it refers to the processing result storage unit 24. Based on all of the recognition result data stored in the processing result storage unit 24 and the supervised data corresponding to each of the recognition result data, the integrated arithmetic formula selection unit 23 calculates, for example, the degree of agreement between the recognition result data and the supervised data corresponding to the recognition result data. The integrated arithmetic formula selection unit 23 detects the combination of the supervised data and the recognition result data that has the greatest calculated degree of agreement. Based on the detected recognition result data, the integrated arithmetic formula selection unit 23 detects conversion parameters and an integrated arithmetic formula to be written to the conversion integration condition storage unit 10. The integrated arithmetic formula selection unit 23 writes and stores the detected conversion parameters and integrated arithmetic formula in the conversion integration condition storage unit 10 (step Sb12).

[0102] For example, the integrated calculation formula selection unit 23 selects a vector y2 that is the correct answer data and a vector ^y 2,2 In this case, the integrated arithmetic formula selection unit 23 detects the combination of the correct answer data and the recognition result data that has the highest degree of match between the vector y1 and the recognition result data. N and the vector ^y 2,2and the integrated arithmetic formula corresponding to the index of "integrated arithmetic formula 2" corresponding to the index are written to the conversion integration condition storage unit 10. When the processing of writing to the conversion integration condition storage unit 10 by the integrated arithmetic formula selection unit 23 is completed, the processing of FIG. 5 is completed.

[0103] The transformation parameters ξ1 to ξ written in the transformation integration condition storage unit 10 N and the integrated arithmetic expression corresponding to the index of "integrated arithmetic expression 1" is the optimal combination for dealing with image degradation occurring in the input data given to the inference device 1. Therefore, by using the transformation integration condition storage unit 10 as the transformation integration condition storage unit 10 of the inference device 1, highly accurate recognition results can be obtained in the inference device 1.

[0104] The combination of multiple conversion parameters selected by the condition selection device 2 and one integrated arithmetic formula is optimized for multiple input data containing common image degradation. Therefore, when input data containing image degradation different from the image degradation of the input data contained in the training data stored in the training data storage unit 21 of the condition selection device 2 is provided to the inference device 1, it is necessary to use the condition selection device 2 to select a new combination of multiple conversion parameters and one integrated arithmetic formula.

[0105] (Another configuration example of the condition selection device of the first embodiment) In the condition selection device 2 of the first embodiment described above, the conversion parameter designation unit 11a selects N conversion parameters ξ1 to ξ2 from the outside. NThe processing of loops Lb2s to Lb2e is performed using all of the above. Alternatively, the transformation parameter designation unit 11a may change the combination of transformation parameters designated to the transformation unit 13 and the inverse transformation unit 15 for each set of training data. For example, the transformation parameter designation unit 11a takes in more than N transformation parameters provided from the outside. Each time a start instruction signal is received from the data reading unit 22, the transformation parameter designation unit 11a randomly selects N transformation parameters, including a transformation parameter ξ1 that does not perform image transformation, from the taken-in transformation parameters. The processing of loops Lb2s to Lb2e may be performed based on each of the N transformation parameters randomly selected by the transformation parameter designation unit 11a.

[0106] In this case, when the number of transformation parameters externally provided to the transformation parameter designation unit 11a is L, the value of N may be set to, for example, about 10% of L. The transformation parameter designation unit 11a may arbitrarily change the number of transformation parameters to be selected for each selection, rather than fixing the number to be randomly selected to N. Incidentally, as described above, in the processing of step Sb5, the transformation parameter designation unit 11a designates the N transformation parameters ξ1 to ξ2 stored in the internal storage area for the integration unit 16a. N On the other hand, if the conversion parameter designation unit 11a changes the combination of designated conversion parameters every time it receives a start instruction signal, the conversion parameter designation unit 11a outputs the number of selected conversion parameters to the integration unit 16a.

[0107] As described above, if the combination of transformation parameters changes for each training data, the transformation parameters stored in the "Transformation Parameters" field of the table in the processing result storage unit 24 in FIG. 6 will change for each record. This makes it possible to select a combination of optimal transformation parameters and integrated arithmetic expressions for multiple input data containing a common image degradation, where the number of transformation parameters is less than the number of externally provided transformation parameters. This reduces the processing load on the inference device 1. Note that although M, the number of integrated arithmetic expressions externally provided to the integrating unit 16a of the condition selection device 2, is an integer greater than or equal to two, if the combination of transformation parameters changes for each training data, M may be set to 1, i.e., the number of integrated arithmetic expressions in the integrating unit 16a may be set to one. When M=1, the condition selection device 2 does not select an integrated arithmetic expression, but rather selects optimal transformation parameters for multiple input data containing a common image degradation.

[0108] In the inference device 1 of the first embodiment, the transformation parameter designation unit 11 designates transformation parameters. The transformation unit 13 performs transformation on input data based on each of the designated transformation parameters to generate transformed data. The reliability map generation unit 14 generates a reliability map, which is data indicating the characteristics of each of the transformed data, for each of the transformed data. The inverse transformation unit 15 performs, based on each of the designated transformation parameters, an inverse transformation of the transformation performed when the corresponding transformed data was generated, on each of the reliability maps to generate inversely transformed data. The integration unit 16 performs an integration process to integrate each of the inversely transformed data, generating integrated data whose number of dimensions matches the reliability map of the input data. The analysis unit 17 performs a recognition process, as an example of an analysis process, on the integrated data. In this way, by generating a reliability map of the transformed data generated by transformation from the input data, inversely transforming and integrating the generated reliability map, and performing a recognition process on the integrated integrated data, it is possible to perform an inference process that is robust against unknown degradation occurring in the input data.

[0109] That is, the inference device 1 according to the first embodiment is configured to perform multiple image transformations on input data to reduce image degradation before performing recognition processing, then extract features, and then inversely transform and integrate a reliability map including the extracted features. In other words, the method adopted by the inference device 1 is a combination of existing methods, namely, image transformation by the transformation unit 13, inverse transformation of the image transformation by the inverse transformation unit 15, ensembling by the integration unit 16, and a semantic segmentation algorithm including downsampling and upsampling by the reliability map generation unit 14 and the analysis unit 17. Although this method combines existing methods, it is an effective method that can reduce the impact of unknown image degradation. This method also allows existing semantic segmentation algorithms to be used as is without retraining. Therefore, the inference device 1 enables robust inference processing against unknown degradation occurring in input data without retraining, such as fine-tuning, the trained semantic segmentation neural network.

[0110] (Second embodiment) 7 is a block diagram showing the configuration of an inference device 1a according to the second embodiment. In the inference device 1a of the second embodiment, the same components as those of the inference device 1 of the first embodiment are given the same reference numerals, and only the different components will be described below. The inference device 1a includes a conversion integration condition storage unit 10, a conversion parameter designation unit 11, a data import unit 12a, a conversion unit 13a, a reliability map generation unit 14, an inverse conversion unit 15, an integration unit 16b, an analysis unit 17, an output unit 18, and a quality evaluation unit 19.

[0111] When data acquisition unit 12a acquires input data provided from the outside, it outputs the acquired input data to conversion unit 13a and quality evaluation unit 19. Quality evaluation unit 19 evaluates the quality of the input data output by data acquisition unit 12a. If the input data is, for example, image data, quality evaluation unit 19 evaluates, for example, the image quality indicated by the image resolution indicating the pixel density of the image data. Quality evaluation unit 19 outputs data indicating the quality as the evaluation result to integration unit 16b.

[0112] The conversion unit 13a has the same configuration as the conversion unit 13 of the first embodiment except for the configuration described below. That is, the conversion unit 13 of the first embodiment outputs converted data generated by image conversion to the reliability map generation unit 14. In contrast, the conversion unit 13a of the second embodiment outputs converted data generated by image conversion to the reliability map generation unit 14 and the integration unit 16b.

[0113] The integrating unit 16b has the same configuration as the integrating unit 16 of the first embodiment except for the configuration described below. That is, in the first embodiment, the integrating unit 16 uses the weights w n When performing integration processing using an integration formula including n In contrast, in the second embodiment, the integration unit 16b uses the weights w n When the integration process is performed using an integration formula including the following, the inverse transformation data vector p n Weight of each w n Determine the value of .

[0114] (Processing by the inference device of the second embodiment) 8 is a flowchart showing the flow of processing by the inference device 1a of the second embodiment. In Fig. 8, the processing of steps Sc1 to Sc3 is the same as the processing of steps Sa1 to Sa3 in Fig. 3, except that data acquisition unit 12 is replaced with data acquisition unit 12a and integrating unit 16 is replaced with integrating unit 16b. In the processing of step Sc3, data acquisition unit 12a acquires input data, which is image data of the recognition target provided from the outside, and outputs the acquired input data to conversion unit 13a and quality evaluation unit 19.

[0115] The processing of loop Lc1s to Lc1e is the same as that of loop La1s to La1e in Fig. 3, except that the conversion unit 13 is replaced with a conversion unit 13a. In the processing of step Sc4 included in the processing of loop Lc1s to Lc1e, the conversion unit 13a outputs converted data generated by image conversion to the reliability map generation unit 14 and the integration unit 16b. The integration unit 16b sequentially takes in the converted data output by the conversion unit 13a during the processing of loop Lc1s to Lc1e.

[0116] In parallel with the processing of loop Lc1s to Lc1e, the quality evaluation unit 19 takes in the input data output by the data acquisition unit 12a and evaluates the quality of the taken-in input data. The quality evaluation unit 19 outputs data indicating the evaluated quality to the integration unit 16b. The integration unit 16b takes in the data indicating the quality output by the quality evaluation unit 19 (step Sc7).

[0117] When one processing of the loop Lc1s to Lc1e is completed, the integration unit 16b converts the vector ~x n and a vector p, which is one piece of inverse transformation data output by the inverse transformation unit 15. n When one processing of the loop Lc1s to Lc1e is performed, the integration unit 16b takes in one vector ~x n and one vector p n means that the vector p n is the vector ~x nTherefore, the integration unit 16b converts the converted data vector ∼x n and the vector p n The combination is associated with the above and written and stored in an internal memory area.

[0118] The integration unit 16b uses the transformation parameters ξ1 to ξ2 stored in the transformation integration condition storage unit 10. N The integration unit 16b refers to "N", which is the number of vectors p n When the number of items reaches N, the weight w n The integration unit 16b determines whether the weight w is included in the integration calculation formula (step Sc8). n If it is determined that the value does not include the specified value (No in step Sc8), the process proceeds to step Sc10.

[0119] On the other hand, the integration unit 16b adds a weight w n If it is determined that the vector p is included (step Sc8, Yes), the inverse transformation data is generated based on the data indicating the quality. n Weight of each w n For example, the integration unit 16b determines the value of (15). Assume that the integration formula read from the conversion integration condition storage unit 10 is the integration formula for the weighted maximum value of equation (15), and the data indicating quality is data indicating image resolution. The integration unit 16b determines whether the image resolution of the input data is high or low based on whether the image resolution of the input data is equal to or higher than a predetermined threshold.

[0120] The integration unit 16b determines that the image quality of the input data is high when the image resolution of the input data is equal to or higher than a predetermined threshold. In this case, the integration unit 16b integrates the vector ~x n The integration unit 16b detects the image resolution of each of the vectors ∼x with high image resolution based on the detected image resolution. nhas a higher ranking than the vector ~x n The integration unit 16b assigns a rank to the vectors ~x n The vector p associated with n The weight of n The vector p n Weight of each w n Determine the value of .

[0121] On the other hand, if the image resolution of the input data is less than a predetermined threshold, the integration unit 16b determines that the image quality of the input data is low. In this case, the integration unit 16b integrates the vector ~x n The integration unit 16b detects the image resolution of each of the vectors ∼x with low image resolution based on the detected image resolution. n has a higher ranking than the vector ~x n The integration unit 16b assigns a rank to the vectors ~x n The vector p associated with n The weight of n The vector p n Weight of each w n In this way, the integration unit 16b determines the value of the weight w included in the integration formula based on the data indicating the quality. n The process of steps Sc10 to Sc12 is the same as the process of steps Sa7 to Sa9 in FIG. 3, except that the integrator 16 is replaced with the integrator 16b.

[0122] With the configuration of the second embodiment described above, the weight w n If it contains, the weight of the integration formula w is determined based on the quality of the input data. n Therefore, it is possible to obtain a more accurate result of the recognition process. In addition, when the image resolution of the input data is high, as described above, the vector ~x n vector p associated with n The weight of nIf the image resolution of the input data is low, increase the value of n vector p associated with n The weight of n The reason for increasing the value of is as follows: if the image quality of the input data is high, the vector ~x with high image resolution is n The vector p associated with n It is expected that the recognition results obtained based on the vector p n Although is robust against image degradation such as noise and blur, it may not be possible to obtain accurate segmentation results for small objects, for example. Therefore, when the image quality of the input data is high, it is recommended to use a vector ~x with a high image resolution. n vector p associated with n The weight of n On the other hand, if the image quality of the input data is low, the value of the low-resolution vector ~x n The vector p associated with n It is expected that the recognition results obtained based on the low-resolution vector ~x n The vector p associated with n is more robust against image degradation such as strong noise and blur. Therefore, when the image quality of the input data is low, the vector ~x with low image resolution is n vector p associated with n The weight of n The value of is increased.

[0123] (Another Configuration Example (Part 1) of the Inference Device of the Second Embodiment) In the second embodiment, the integration unit 16b determines whether the image resolution of the input data is high or low based on one predetermined threshold. Alternatively, two thresholds, a first threshold and a second threshold, may be predetermined so that the first threshold is greater than the second threshold, and the following processing may be performed. When the image resolution of the input data is equal to or greater than the first threshold, the integration unit 16b determines that the image quality of the input data is high, and the vector p n Weight of each w n When the image resolution of the input data is less than a predetermined second threshold, the integration unit 16b determines that the image quality of the input data is low, and performs a process of determining the value of the vector p n Weight of each w n When the image resolution of the input data is less than the first threshold value and equal to or greater than the second threshold value, the integration unit 16 performs a process of determining the value of the predetermined weight w n applies.

[0124] (Another Configuration Example (Part 2) of the Inference Device of the Second Embodiment) In the second embodiment, the integration unit 16b integrates the data indicating the quality of the input data output by the quality evaluation unit 19 and the vector ∼x n and based on the weights w n In response to this, the integration unit 16 determines the value of the data indicating the quality of the input data output by the quality evaluation unit 19 and the conversion parameter ξ n and based on the weights w n The value of the transformation parameter ξ may be determined. n If the contents of are known, the transformation parameter ξ n This is because it is possible to calculate the relative image resolution of the converted data obtained by image conversion using the above method. In this case, the inference device 1a needs to be configured as follows.

[0125] In the inference device 1a, the conversion unit 13a is replaced with the conversion unit 13 of the first embodiment. When the conversion parameter designation unit 11 outputs the conversion parameters to the conversion unit 13 and the inverse conversion unit 15 in the processing of the loop Lc1s to Lc1e, the conversion parameter designation unit 11 also outputs the conversion parameter ξ to the integration unit 16b. n The integration unit 16b outputs one transformation parameter ξ output by the transformation parameter designation unit 11 for each process of the loops Lc1s to Lc1e. n and a vector p, which is one piece of inverse transformation data output by the inverse transformation unit 15. n The integration unit 16b takes in one of the transformation parameters ξ n and one inverse transformed data vector p n The data is written and stored in an internal memory area in association with the above.

[0126] In the process of step Sc9, the integration unit 16b calculates the transformation parameters ξ stored in the internal storage area. n Based on each of the transformation parameters ξ n When the image quality of the input data is determined to be high, the integration unit 16b calculates the relative image resolution of the converted data obtained when the conversion unit 13 performs image conversion using the conversion parameter ξ n has a higher rank than the transformation parameter ξ n The integration unit 16b assigns a rank to the transformation parameters ξ n The vector p associated with n The weight of n The vector p n Weight of each w n On the other hand, when the integration unit 16b determines that the image quality of the input data is low, it determines the value of the conversion parameter ξ n has a higher rank than the transformation parameter ξ n The integration unit 16b assigns a rank to the transformation parameters ξ n The vector p associated with n The weight of nThe vector p n Weight of each w n Determine the value of .

[0127] (Other supplementary configurations in the second embodiment) The configuration of the other configuration example (part 1) of the inference device of the second embodiment may be applied to the configuration of the other configuration example (part 2) of the inference device of the second embodiment. In the second embodiment, the quality evaluation unit 19 may evaluate quality other than image resolution. In the second embodiment, the integration unit 16b evaluates the weight w n When defining the vector ~x, which is the transformation data, n and calculates an index other than the image resolution from the weight w based on the calculated index and the data indicating the quality of the input data output by the quality evaluation unit 19. n In the second example of the second embodiment, the integration unit 16b may determine the transformation parameter ξ n and calculates an index other than the relative image resolution from the weight w based on the calculated index and the data indicating the quality of the input data output by the quality evaluation unit 19. n may be determined.

[0128] (Third embodiment) FIG. 9 is a block diagram showing the configuration of a conversion parameter selection device 3 according to the third embodiment. The conversion parameter selection device 3 is a device used together with, for example, the condition selection device 2 according to the first embodiment. It is assumed that among the multiple conversion parameters written into the conversion integration condition storage unit 10 by the integration arithmetic expression selection unit 23 of the condition selection device 2 according to the first embodiment, there may be conversion parameters that contribute little to improving the accuracy of the recognition processing by the analysis unit 17. By excluding such conversion parameters with low contribution, it is possible to reduce the processing load performed by the inference device 1. The conversion parameter selection device 3 is a device that detects such conversion parameters with low contribution, deletes the detected conversion parameters from the multiple conversion parameters stored in the conversion integration condition storage unit 10, and selects multiple conversion parameters to be stored in the conversion integration condition storage unit 10.

[0129] In the conversion parameter selection device 3 of the third embodiment, the same components as those in the inference device 1 and condition selection device 2 of the first embodiment are assigned the same reference numerals, and only the different components will be described below. The conversion parameter selection device 3 includes a conversion integration condition storage unit 10, a conversion parameter designation unit 11b, a conversion unit 13, a reliability map generation unit 14, an inverse conversion unit 15, an analysis unit 17, a training data storage unit 21a, a data reading unit 22, a processing result storage unit 24a, and a conversion parameter selection unit 25. The data reading unit 22 included in the condition selection device 2 of the first embodiment is connected to the training data storage unit 21, the conversion parameter designation unit 11a, the conversion unit 13, and the integration arithmetic expression selection unit 23. In contrast, the data reading unit 22 provided in the conversion parameter selection device 3 of the third embodiment is connected to the conversion unit 13, connected to the training data storage unit 21a instead of the training data storage unit 21, connected to the conversion parameter designation unit 11b instead of the conversion parameter designation unit 11a, and connected to the conversion parameter selection unit 25 instead of the integrated arithmetic expression selection unit 23.

[0130] The conversion integration condition storage unit 10 included in the conversion parameter selection device 3 is the conversion integration condition storage unit 10 in the condition selection device 2 of the first embodiment, into which a plurality of conversion parameters and one integrated arithmetic expression are written by the integrated arithmetic expression selection unit 23. Here, the plurality of conversion parameters stored in the conversion integration condition storage unit 10 are, for example, N conversion parameters ξ1 to ξ N Let us assume that:

[0131] The conversion parameter designation unit 11b designates N conversion parameters ξ1 to ξ2 stored in the conversion integration condition storage unit 10. N The transformation parameter designation unit 11b reads out the N transformation parameters ξ1 to ξ N Upon receiving the start instruction signal, the transformation parameter designation unit 11b generates N indexes of "transformation parameter 1", ..., "transformation parameter N" for each of the transformation parameters ξ1 to ξ N One transformation parameter ξ nThe conversion parameter designation unit 11b selects the selected conversion parameter ξ n to the transform unit 13 and the inverse transform unit 15, the transform parameters ξ n The conversion parameter designation unit 11b further designates the selected conversion parameter ξ n The index of the “transformation parameter n” corresponding to the parameter n is output to the transformation parameter selection unit 25.

[0132] The training data storage unit 21a stores a plurality of training data in advance. In the first embodiment, all of the generated plurality of training data are stored in the training data storage unit 21 of the condition selection device 2. In contrast, in the third embodiment, a portion of the generated plurality of training data is selected as validation data, and the training data selected as validation data is stored in the training data storage unit 21a of the conversion parameter selection device 3 without being stored in the training data storage unit 21 of the condition selection device 2. The processing result storage unit 24a stores N pieces of recognition result data output by the analysis unit 17 to the conversion parameter selection unit 25 for each piece of supervised answer data output by the data reading unit 22 to the conversion parameter selection unit 25.

[0133] The conversion parameter selection unit 25 selects the conversion parameters ξ1 to ξ2 stored in the conversion integration condition storage unit 10 for one training data. N When N pieces of recognition result data that match N, the number of recognition result data, are obtained, the conversion parameter selection unit 25 outputs a continuation instruction signal to the data reading unit 22. Upon receiving the termination instruction signal, the conversion parameter selection unit 25 selects conversion parameters to be excluded from the plurality of conversion parameters based on all of the recognition result data obtained up until receiving the termination instruction signal and the correct answer data corresponding to each of the recognition result data. The conversion parameter selection unit 25 deletes the selected conversion parameters from the conversion integration condition storage unit 10. The conversion integration condition storage unit 10 after the conversion parameters have been deleted by the conversion parameter selection unit 25 will be used as the conversion integration condition storage unit 10 of the inference device 1 of FIG. 1.

[0134] (Processing of the conversion parameter selection device of the third embodiment) 10 is a flowchart showing the flow of processing by the conversion parameter selection device 3 of the third embodiment. As a prerequisite for starting the flowchart shown in FIG. 10, a plurality of training data selected as validation data is written in advance to the training data storage unit 21 of the conversion parameter selection device 3. The conversion integration condition storage unit 10 stores N conversion parameters ξ1 to ξ2 written by the integration arithmetic expression selection unit 23 of the condition selection device 2. N One integrated arithmetic expression is written in. The processing result storage unit 24a is initialized and is set to a state in which no data is stored.

[0135] When the conversion parameter selection device 3 is started, the conversion parameter designation unit 11b selects N conversion parameters ξ1 to ξ2 stored in the conversion integration condition storage unit 10. N The transformation parameter designation unit 11b reads out the N transformation parameters ξ1 to ξ N The transformation parameter specifying unit 11b generates indexes of N "transformation parameters 1", ..., "transformation parameters N" corresponding to the N transformation parameters ξ1 to ξ N and N transformation parameters ξ1 to ξ N The conversion parameter specification unit 11b associates the generated N indices of "conversion parameter 1", ..., "conversion parameter N" with each of the N indices and writes and stores them in an internal storage area. The conversion parameter specification unit 11b outputs the generated indices of "conversion parameter 1", ..., "conversion parameter N" to the conversion parameter selection unit 25 (step Sd1).

[0136] The conversion parameter selection unit 25 takes in the indexes of N "conversion parameter 1", ..., "conversion parameter N". The conversion parameter selection unit 25 generates, in the processing result storage unit 24a, a table in the data format shown in Fig. 11, which has an item of "correct answer data" and items of N "conversion parameter 1", ..., "conversion parameter N" corresponding to each of the indexes of the taken N "conversion parameter 1", ..., "conversion parameter N" (step Sd2).

[0137] When the data reading unit 22 receives a start instruction signal from the outside (step Sd3), the first process of the loop Ld1s to Ld1e is started. The data reading unit 22 reads one piece of training data from the training data storage unit 21a. After reading the training data, the data reading unit 22 outputs a start instruction signal to the conversion parameter designation unit 11b. The data reading unit 22 outputs the input data included in the read training data to the conversion unit 13. The conversion unit 13 takes in the input data output by the data reading unit 22. The data reading unit 22 outputs the correct answer data included in the read training data to the conversion parameter selection unit 25 (step Sd4).

[0138] The conversion parameter selection unit 25 takes in the correct data output by the data reading unit 22. Here, the vector y, which is the correct data taken in by the conversion parameter selection unit 25, is written as vector y1, vector y2, ... with a subscript indicating the order in which it was taken in. When the conversion parameter selection unit 25 takes in vector y1, which is the first correct data, it generates one record in the table of the processing result storage unit 24a and writes the taken in vector y1 in the "correct data" field of the generated record (step Sd5).

[0139] When the conversion parameter designation unit 11b receives a start instruction signal from the data reading unit 22, it reads N conversion parameters ξ1 to ξ2 stored in an internal storage area. N One of the transformation parameters ξ n The conversion parameter designation unit 11b further selects and reads out the read conversion parameter ξ from the internal storage area. n The transformation parameter designation unit 11b reads out the index of the "transformation parameter n" associated with the read transformation parameter ξ nto the conversion unit 13 and the inverse conversion unit 15, and outputs the index of the read "conversion parameter n" to the conversion parameter selection unit 25. This starts the first processing of the loop Ld2s to Ld2e. In the processing included in the loop Ld2s to Ld2e, the processing of steps Sd6 to Sd8 is the same as the processing of steps Sa4 to Sa6 in FIG. 3.

[0140] In the process of step Sd8, the inverse transformation unit 15 converts the generated inverse transformation data, that is, the vector p n The analysis unit 17 outputs the vector p n and the resulting vector p n The analysis unit 17 performs a recognition process by applying a function g(·) shown in equation (4) to generate recognition result data. The analysis unit 17 outputs the vector ^y, which is the recognition result data, to the transformation parameter selection unit 25 (step Sd9).

[0141] The conversion parameter selection unit 25 takes in the index of the "conversion parameter n" output by the conversion parameter specification unit 11b and the recognition result data output by the analysis unit 17. The conversion parameter selection unit 25 writes the taken-in recognition result data into the "conversion parameter n" field corresponding to the index of the taken-in "conversion parameter n" in the most recently generated record, i.e., the record in which the "correct answer data" field in the table of the processing result storage unit 24 is the vector y1. Here, when n=2, as shown in FIG. 11, the vector ^y 1,2 is written in the "Conversion Parameter 2" field (step Sd10).

[0142] The conversion parameter designation unit 11b designates the conversion parameters ξ1 to ξ2 stored in an internal storage area. NThen, one unselected transformation parameter is read out from the list, and an index corresponding to that transformation parameter is read out. Then, the processing of steps Sd6 to Sd10 is performed again (loop Ld2s to Ld2e). Note that the processing of step Sd5 and the processing of loop Ld2s to Ld2e are both performed in parallel, since they are both processes that start when the data reading unit 22 outputs the input data and the correct answer data in the processing of step Sd4.

[0143] N transformation parameters ξ1 to ξ N When the processing of steps Sd6 to Sd10 corresponding to each of the vectors y1 and y2 is completed, the processing of loops Ld2s to Ld2e is completed. When the conversion parameter specifying unit 11b writes the recognition result data into all of the items "conversion parameter 1", ..., "conversion parameter N" of the record corresponding to vector y1 in the table of the processing result storage unit 24a, it outputs a continuation instruction signal to the data reading unit 22.

[0144] When the data reading unit 22 receives a continue instruction signal from the conversion parameter designation unit 11b, it reads from the training data storage unit 21a any one of the training data that has not been read as a processing target up to that point. After reading the training data, the data reading unit 22 outputs a start instruction signal to the conversion parameter designation unit 11b. As a result, the processing of steps Sd4 and Sd5 and loop Ld2s to Ld2e is performed on the training data read by the data reading unit 22 (loop Ld1s to Ld1e). When all of the training data stored in the training data storage unit 21a has been read as a processing target, the data reading unit 22 cannot read any more training data, so it does not output a start instruction signal to the conversion parameter designation unit 11b and outputs an end instruction signal to the conversion parameter selection unit 25.

[0145] When the conversion parameter selection unit 25 receives an end instruction signal from the data reading unit 22, it refers to the processing result storage unit 24a. Based on all the recognition result data stored in the processing result storage unit 24a and the supervised answer data corresponding to each of the recognition result data, the conversion parameter selection unit 25 calculates, for example, the degree of agreement between the recognition result data and the supervised answer data corresponding to the recognition result data. For example, the conversion parameter selection unit 25 uses a numerical value indicating the calculated degree of agreement as the conversion parameters ξ1 to ξ N The sum is calculated as the transformation parameters ξ1 to ξ N The conversion parameter selection unit 25 selects conversion parameters whose value indicating the degree of match is less than a predetermined threshold as conversion parameters that contribute little to improving the accuracy of the recognition process by the analysis unit 17. The conversion parameter selection unit 25 deletes the selected conversion parameters from the conversion integration condition storage unit 10 (step Sd11) and ends the process. Note that the conversion parameter selection unit 25 may select a predetermined number of conversion parameters in ascending order of the value indicating the degree of match as conversion parameters that contribute little to improving the accuracy of the recognition process by the analysis unit 17.

[0146] This allows conversion parameters that have little contribution to improving the accuracy of the recognition processing by the analysis unit 17 to be deleted from the conversion integration condition storage unit 10, thereby reducing the processing load of the inference device 1 that performs processing based on multiple conversion parameters stored in the conversion integration condition storage unit 10.

[0147] (Other supplementary configurations) In the first and second embodiments described above, in the inference devices 1, 1a and the condition selection device 2, the conversion parameter ξ1 that does not perform image conversion is included in the conversion parameters that are output to the conversion units 13, 13a and the inverse conversion unit 15. In contrast to this, the conversion parameter ξ1 that does not perform image conversion may not be included in the conversion parameters that are output to the conversion units 13, 13a and the inverse conversion unit 15. In other words, the N conversion parameters ξ1 to ξN All of the N transformation parameters ξ1 to ξ2 may be used as transformation parameters for performing image transformation. N All of the N transformation parameters ξ1 to ξ may be used as transformation parameters for image transformation. N When all of the above are used as conversion parameters for performing image conversion, N may be set to 1. That is, one conversion parameter stored in advance in the conversion integration condition storage unit 10 of the inference device 1, 1a may be used as the conversion parameter for performing image conversion. A conversion parameter externally provided to the conversion parameter designation unit 11a of the condition selection device 2 may be used as the conversion parameter for performing one image conversion. When the conversion parameter designation unit 11a shown in the other configuration example of the first embodiment described above selects conversion parameters randomly, the conversion parameter designation unit 11a may select multiple conversion parameters for performing image conversion, or may select any one conversion parameter for performing image conversion.

[0148] Although the training data storage units 21 and 21a are described above as storing a plurality of pieces of training data, the number of pieces of training data stored in the training data storage units 21 and 21a may be one.

[0149] In the inference devices 1, 1a of the first and second embodiments described above, the conversion parameter designation unit 11 selects all conversion parameters stored in the conversion integration condition storage unit 10 in the processing of the loops La1s to La1e in Fig. 3 and the loops Lc1s to Lc1e in Fig. 8. Alternatively, the conversion parameter designation unit 11 may randomly select a predetermined number of conversion parameters or an arbitrarily determined number of conversion parameters from among the conversion parameters stored in the conversion integration condition storage unit 10, and perform the processing of the loops La1s to La1e in Fig. 3 and the loops Lc1s to Lc1e in Fig. 8. In this case, when the conversion parameter designation unit 11 randomly selects conversion parameters, it is necessary to output the number of selected conversion parameters to the integration unit 16, 16b. The integration unit 16, 16b determines whether all inverse conversion data has been imported based on the number of conversion parameters output by the conversion parameter designation unit 11, not based on the number of conversion parameters stored in the conversion integration condition storage unit 10.

[0150] The quality evaluation unit 19 included in the inference device 1a of the second embodiment may be applied to the condition selection device 2 of the first embodiment. In this case, the quality evaluation unit 19 is connected to the data reading unit 22 and the integrating unit 16a of the condition selection device 2, and when the data reading unit 22 outputs input data included in the training data to the converting unit 13, the data reading unit 22 also outputs the input data to the quality evaluation unit 19. The converting unit 13 is replaced with the converting unit 13a, and the converting unit 13a outputs converted data to the reliability map generating unit 14 and the integrating unit 16a. The integrating unit 16a, like the integrating unit 16b of the second embodiment, uses the weights w n When the integration process is performed using an integration formula including the following, the weight w is calculated based on the data indicating the quality output by the quality evaluation unit 19. n The value of is determined. The above-mentioned "Other configuration examples of the inference device 1a of the second embodiment" (parts 1) and (part 2) and "Other supplementary configurations in the second embodiment" may be applied to the condition selection device 2 of the first embodiment to which the quality evaluation unit 19 is applied.

[0151] The condition selection device 2 of another configuration example of the first embodiment and the conversion parameter selection device 3 of the third embodiment may be used together. In this case, the condition selection device 2 of another configuration example of the first embodiment randomly selects conversion parameters and selects conversion parameters that are optimal for reducing image degradation of the input data from the combinations of randomly selected conversion parameters. This prevents the statistical nature of randomly selecting conversion parameters, i.e., the creation of biased combinations of conversion parameters, while reducing the processing load of the inference devices 1 and 1a. In contrast, the selection of conversion parameters performed by the conversion parameter selection device 3 is different in that it reduces the processing load of the inference devices 1 and 1a by eliminating conversion parameters that have a low contribution to improving the accuracy of the recognition process by the analysis unit 17.

[0152] In the first to third embodiments and other configuration examples described above, the input data is three-dimensional data having vertical, horizontal, and channel directions, in other words, two-dimensional image data. However, any data generally applicable to machine learning may be used as the input data. For example, the input data may be one-dimensional time-series data such as voice, sound, or vibration, four- or more-dimensional data such as three-dimensional image data, or symbolic data such as natural language.

[0153] The correct answer data included in the training data stored in the training data storage units 21 and 21a is, for example, a class in semantic segmentation. In contrast, any data that generally serves as a target variable when machine learning is applied may be used as the correct answer data. For example, the correct answer data may be data obtained by digitizing data corresponding to a class or label, such as an attribute, a class name, or a person ID (Identification), or may be data of the same type as the input data. Making the correct answer data the same type as the input data means that if the input data is image-type data, the correct answer data is also image-type data. Generally, in machine learning, when data corresponding to a class or label is used as a target variable, the objective is identification or authentication, and when data of the same type as the input data is used as a target variable, the objective is regression.

[0154] The reliability map generation unit 14 and the analysis unit 17 described above are configured to perform processing based on a semantic segmentation algorithm. Alternatively, the reliability map generation unit 14 and the analysis unit 17 may perform processing based on an algorithm that uses a trained neural network other than the semantic segmentation algorithm, or may perform processing based on an algorithm that does not use a neural network but is other than the semantic segmentation algorithm.

[0155] The above-described analysis unit 17 performs recognition processing, for example, image segmentation processing that divides image regions into classes, which is performed in a semantic segmentation algorithm. However, the analysis unit 17 may perform recognition processing other than image segmentation processing that divides image regions into classes, which is performed in a semantic segmentation algorithm, or may perform any type of analysis processing other than recognition, such as detection, identification, authentication, regression, etc.

[0156] The inference device may be an apparatus in which the inference device 1 of the first embodiment and the condition selection device 2 of the first embodiment are integrated together and common functional units are shared. The inference device may be an apparatus in which the inference device 1 of the first embodiment, the condition selection device 2 of the first embodiment, and the conversion parameter selection device 3 of the third embodiment are integrated together and common functional units are shared. In these integrations, instead of the inference device 1 of the first embodiment, any of the other configuration examples of the inference device 1, such as the inference device 1a of the second embodiment, may be the target of integration. In these integrations, instead of the condition selection device 2 of the first embodiment, any of the other configuration examples of the condition selection device 2, such as the condition selection device 2 of the other configuration example of the first embodiment, may be the target of integration.

[0157] In the configurations of the second and third embodiments described above, a determination process using an inequality sign with an equal sign is performed in the process of step Sc9 performed by the integration unit 16b and the process of step Sd11 performed by the conversion parameter selection unit 25. However, the present invention is not limited to these embodiments, and the determination processes of "greater than or equal to" and "less than or equal to" are merely examples, and may be replaced with determination processes of "exceeding or not" and "equal to or less than" respectively, depending on how the thresholds are defined.

[0158] The three devices, i.e., the inference device 1, 1a, the condition selection device 2, and the conversion parameter selection device 3 in the above-described embodiments, as well as other configuration examples and supplementary configurations of the three devices, may be realized by a computer. In this case, in the inference devices 1, 1a shown in FIGS. 1 and 7, the data acquisition units 12, 12a do not acquire input data provided from outside, but a storage unit that stores multiple pieces of input data in advance may be provided inside or outside the inference device 1, and the data acquisition units 12, 12a may read input data one by one from the storage unit and output the read input data to the conversion unit 13, 13a. The output unit 18 may be provided outside the inference device 1. In the condition selection device 2 shown in FIG. 4, the training data storage unit 21 and the processing result storage unit 24 may be provided outside the condition selection device 2. In the conversion parameter selection device 3 shown in FIG. 9, the training data storage unit 21a and the processing result storage unit 24a may be provided outside the conversion parameter selection device 3. 1 and 7, the condition selection device 2 shown in Fig. 4, and the conversion parameter selection device 3 shown in Fig. 9, the conversion integration condition storage unit 10 may be provided as a storage device outside each device, or each device may be connected to share the external storage device. Programs for realizing the functions of the three devices, i.e., the inference devices 1 and 1a, the condition selection device 2, and the conversion parameter selection device 3, as well as other configuration examples of the above three devices and configurations shown supplementarily, may be recorded on a computer-readable recording medium, and the programs recorded on this recording medium may be read into a computer system and executed.

[0159] The term "computer system" as used herein includes hardware such as an operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or over communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may also be designed to implement some of the functions described above, or may be capable of implementing the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0160] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0161] 1...inference device, 10...conversion integration condition storage unit, 12...data acquisition unit, 13...conversion unit, 14...reliability map generation unit, 15...inverse conversion unit, 16...integration unit, 17...analysis unit, 18...output unit

Claims

1. a conversion parameter specifying step of specifying one or more conversion parameters; a conversion step of converting input data based on each of the specified conversion parameters to generate converted data; a reliability map generation step of generating, for each of the transformed data, a reliability map which is data indicating each feature of the transformed data; an inverse transformation step of performing an inverse transformation of the transformation performed when the corresponding transformation data was generated for each of the reliability maps based on each of the specified transformation parameters to generate inverse transformation data; an integration step of performing an integration process of integrating each of the inverse transformed data to generate integrated data having a number of dimensions that is equal to the number of dimensions of the reliability map of the input data; an analysis step of performing an analysis process on the integrated data; Including, the analyzing step performs a recognition process as the analysis process, The integration step includes: An inference method in which the integration process is performed using any one optimal integrated arithmetic formula selected from a plurality of integrated arithmetic formulas based on the recognition result obtained by the recognition process in the analysis step performed on the integrated data generated from the input data to which correct answer data has been assigned, and the correct answer data corresponding to the integrated data.

2. The integrated calculation formula is: is an equation for performing a calculation to calculate a feature value at a position of the integrated data based on feature values ​​at a corresponding position in each of the inverse transformed data to be integrated, The inference method of claim 1 .

3. The integrated calculation formula is: an equation for performing a calculation to calculate a feature value at a position of the integrated data based on feature values ​​at a corresponding position in each of the inverse transformed data to be integrated and at positions near the corresponding position; The inference method of claim 1 .

4. further comprising a quality evaluation step of evaluating the quality of the input data; The integration step includes: When the integration formula used in the integration process includes an operation using a weight applied to each of the inverse transformed data, the value of the weight is determined based on data indicating the quality obtained by evaluating the input data corresponding to the inverse transformed data in the quality evaluation step. The inference method of claim 1 .

5. The conversion parameter designation step includes: designating the conversion parameters to be selected based on the recognition results obtained by the recognition processing in the analysis step, which is performed on each of the inverse-converted data obtained from the input data to which the supervised data has been assigned, and the supervised data corresponding to the inverse-converted data; The inference method according to any one of claims 1 to 4.

6. a conversion parameter specification section for specifying one or more conversion parameters; a conversion unit that converts input data based on each of the specified conversion parameters to generate converted data; a reliability map generating unit that generates, for each of the converted data, a reliability map that is data indicating each feature of the converted data; an inverse transformation unit that performs an inverse transformation of a transformation performed when the corresponding transformation data was generated on each of the reliability maps based on each of the designated transformation parameters to generate inverse transformation data; an integration unit that performs integration processing to integrate each of the inverse transformed data and generates integrated data having a number of dimensions that is equal to the number of dimensions of the reliability map of the input data; an analysis unit that performs analysis processing on the integrated data; Equipped with the analysis unit performs a recognition process as the analysis process, The integration unit An inference device that performs the integration process using any one optimal integrated arithmetic formula selected from a plurality of integrated arithmetic formulas based on the recognition result obtained by the recognition process performed by the analysis unit on the integrated data generated from the input data to which correct answer data has been added, and the correct answer data corresponding to the integrated data.

7. On the computer, specifying one or more transformation parameters; a step of performing a transformation on input data based on each of the specified transformation parameters to generate transformed data; a step of generating a reliability map, which is data indicating each feature of the transformed data, for each of the transformed data; a step of performing an inverse transformation of the transformation performed when the corresponding transformation data was generated on each of the reliability maps based on each of the designated transformation parameters to generate inverse transformation data; a step of performing an integration process of integrating each of the inverse transformed data to generate integrated data having a number of dimensions that is equal to the number of dimensions of the reliability map of the input data; performing analytical processing on the integrated data; Execute In the step of performing the analysis processing, a recognition processing is performed as the analysis processing, In the step of generating the integrated data, A program for performing the integration process using any one optimal integration formula selected from a plurality of integration formulas based on the recognition result obtained by the recognition process performed on the integrated data generated from the input data to which correct answer data has been added and the correct answer data corresponding to the integrated data.

Images