Information processing system, information processing device, information processing method, and program

Abstract

The present invention: sets, as a condition, a third image feature for recognition of a subject extracted from an original image; determines, from a conditional reliability of a first image feature identified by using a first machine learning model, a parameter set for the first machine learning model such that a first loss function, indicating the degree of change to a conditional reliability of a second image feature identified in a feature region of a restoration image with the third image feature as a condition, becomes greater; and determines a parameter set for each of a second machine learning model and a third machine learning model such that a second loss function, obtained by synthesizing the conditional reliability of the second image feature with the third image feature as a condition and a feature loss function indicating the degree of change from the third image feature to a fourth image feature for recognition of the subject extracted from the restoration image, becomes smaller.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program. [Background technology]

[0002] Image compression technology is a method of converting an image into compressed data with less information so that the original image can be restored. Image compression technology has a wide range of applications, including image transmission and storage. Image compression technology is applied, for example, to remote monitoring systems. A remote monitoring system includes, for example, an edge device and a data center. The edge device captures images showing the shapes of various objects in a monitored area, compresses the information content of the captured images, converts them into compressed data, and transmits them to the data center. The data center restores the compressed data received from the edge device to a restored image and performs image recognition to detect objects in the monitored area. The data center further displays a monitoring screen showing the detected objects and the restored image of the monitored area.

[0003] With the advancement of artificial intelligence (AI) technology, machine learning models are being applied to image compression. For example, Patent Documents 1 and 2 describe image compression technology that applies generative adversarial networks (GANs). In training a machine learning model, image recognition is performed on the restored image, and a constraint is imposed that the recognition rate must not drop as much as possible compared to the recognition rate for the original image. By using model parameters obtained through training, it is expected that the recognition rate will be improved compared to when no constraints are imposed. Non-Patent Document 1 also describes image compression technology that applies GANs. In the method described in Non-Patent Document 1, a classifier uses a segmented image that shares semantics with the original image data as the classification target, and determines parameter sets for the encoder and generator through training. By maintaining the characteristics of a specified image as sharing the same semantics with the original image, quantitative improvement of the restored image is achieved. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 1,104,8974 [Patent Document 2] U.S. Patent No. 10,944,996 [Non-patent literature]

[0005] [Non-Patent Document 1] Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool, “Generative Adversarial Networks for Extreme Learned image Compression”, International Conference on Computer Vision (ICCV 2019), October 27-November 2, 2019 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the subjective quality obtained by visually inspecting a restored image is not necessarily good. For example, noise patterns such as block noise may become apparent in the restored image. Even if a high recognition rate is achieved by performing image recognition processing on the restored image, the subjective quality may actually be degraded.

[0007] An object of the present invention is to provide an information processing system, an information processing device, an information processing method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0008] According to a first aspect of the present invention, an information processing system includes a first identification means for identifying a first image feature in a feature region of an original image using a first machine learning model for the original image, a compression means for generating compressed data with a reduced data volume using a second machine learning model for the original image, a restoration means for generating a restored image of the original image from the compressed data using a third machine learning model, a second identification means for identifying a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image, a third image feature extraction means for extracting a third image feature for object recognition from the original image, and a fourth image feature extraction means for extracting a fourth image feature for object recognition from the restored image. and a model learning means for determining the parameter sets of the fourth machine learning model and the first machine learning model in common, so that a first loss function indicating the degree of variation from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature becomes larger, and for determining the parameter sets of each of the second machine learning model and the third machine learning model in such a way that a second loss function obtained by combining the conditional reliability of the second image feature conditioned on the third image feature and a feature loss function indicating the degree of variation from the third image feature to the fourth image feature becomes smaller.

[0009] According to a second aspect of the present invention, an information processing method is an information processing method in an information processing system, comprising: a first identification step of identifying a first image feature in a feature region of an original image using a first machine learning model for the original image; a compression step of generating compressed data with a reduced data volume using a second machine learning model for the original image; a restoration step of generating a restored image of the original image from the compressed data using a third machine learning model; a second identification step of identifying a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image; a third image feature extraction step of extracting a third image feature for object recognition from the original image; and a fourth image feature extraction step of extracting a fourth image feature for object recognition from the restored image. The method includes a fourth image feature extraction step, and a model learning step of setting a parameter set of the fourth machine learning model in common with a parameter set of the first machine learning model, determining a parameter set of the first machine learning model so that a first loss function indicating the degree of variation from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature becomes larger, and determining parameter sets of the second machine learning model and the third machine learning model so that a second loss function obtained by combining the conditional reliability of the second image feature conditioned on the third image feature and a feature loss function indicating the degree of variation from the third image feature to the fourth image feature becomes smaller.

[0010] According to a third aspect of the present invention, there is provided an information processing method in an information processing device, comprising: a model learning means for determining a parameter set for the first machine learning model so that a first loss function indicating the degree of variation in the conditional reliability conditional on the third image feature becomes larger, from a conditional reliability of a first image feature identified in a feature region of the original image using a first machine learning model for the original image, using a third image feature for recognizing a subject extracted from the original image as a condition; and a parameter learning means for determining respective parameter sets for the second machine learning model and the third machine learning model so that a second loss function obtained by combining the conditional reliability of the second image feature conditional on the third image feature and a feature loss function indicating the degree of variation from the third image feature to a fourth image feature for recognizing the subject extracted from the restored image becomes smaller. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the subjective quality of a restored image and the recognition rate of image recognition for the restored image. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic block diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a schematic block diagram illustrating an example of the configuration of a compression unit according to the first embodiment. [Figure 3] 3 is a schematic block diagram showing an example of the configuration of a restoration unit according to the first embodiment. FIG. [Figure 4] FIG. 10 is an explanatory diagram for explaining learning of a classifier. [Figure 5] FIG. 10 is an explanatory diagram for explaining the learning of a generator. [Figure 6]5 is a flowchart showing an example of image compression and decompression processing according to the first embodiment. [Figure 7] 5 is a flowchart showing an example of a model learning process according to the first embodiment. [Figure 8] FIG. 2 is a schematic block diagram showing an application example of the information processing system according to the first embodiment. [Figure 9] FIG. 10 is a schematic block diagram showing an example of the functional configuration of a third image feature extraction unit according to the second embodiment. [Figure 10] FIG. 10 is a schematic block diagram showing an example of the configuration of a first discrimination unit according to a second embodiment. [Figure 11] FIG. 10 is a schematic block diagram illustrating an example of the configuration of an information processing system according to a third embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of distribution of image features. [Figure 13] FIG. 10 is a diagram illustrating an example of a recognition rate for a restored image. [Figure 14] FIG. 10 is a diagram showing a first example of a restored image. [Figure 15] FIG. 10 is a diagram showing a second example of a restored image. [Figure 16] FIG. 1 is a schematic block diagram illustrating an example of a minimum configuration of an information processing system. [Figure 17] FIG. 1 is a schematic block diagram illustrating an example of a minimum configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First Embodiment A first embodiment will be described. FIG. 1 is a schematic block diagram showing an example of the configuration of an information processing system 1 according to this embodiment. The information processing system 1 acquires image data representing an image (original image), compresses the data volume of the acquired image data, and generates compressed data. The information processing system 1 expands the data volume of the generated compressed data, and generates reconstructed data representing a reconstructed image of the original image. The information processing system 1 extracts an image feature (referred to as a "fourth image feature" in this application) from the reconstructed image. The information processing system 1 performs image recognition processing, for example, using the extracted fourth image feature.

[0014] The information processing system 1 includes an input processing unit 14, a compression processing unit 30, a first classification unit 32, a second classification unit 34, a third image feature extraction unit 38, a fourth image feature extraction unit 39, and a model learning unit 36. The specific configuration of each of these units is as follows: The compression processing unit 30 includes an encoding unit 12 and a decoding unit 22. The information processing system 1 may be configured as a distributed processing system in which multiple devices are distributed at different spatial locations. For example, the information processing system 1 may be configured to include an edge device (not shown) and a data center (not shown). In the example shown in FIG. 1, one or more functional units may be arranged in each of the individual areas separated by dashed-dotted lines. The location or time may differ for each individual area.

[0015] As described above, when the information processing system 1 is configured as a distributed processing system including edge devices and a data center, the edge devices are installed near the sources of information to be processed and provide computing resources for the information. In the example shown in FIG. 1, image data corresponds to the information to be processed. The edge devices may be configured to include, for example, an input processing unit 14 and an encoding unit 12. The number of edge devices in the information processing system 1 is not limited to one, and may be two or more. Each edge device may further be connected to an imaging unit 16 (described below) via at least one of a wireless and a wired connection.

[0016] On the other hand, the data center executes processing related to the entire distributed processing system using various information provided by the edge devices. The data center may be installed in a location spatially separated from the edge devices. The data center is communicatively connected to each edge device via a network via at least one of wireless and wired connections. The data center includes, for example, a decoding unit 22 and an image recognition unit 42. The data center may further include a first identification unit 32, a second identification unit 34, a third image feature extraction unit 38, a fourth image feature extraction unit 39, and a model learning unit 36.

[0017] The data center may be configured as a single device, but is not limited to this. The data center may include multiple devices and be configured as a cloud that can send and receive data between them. The data center may be configured to include, for example, a server device and a model learning device. The server device may include, for example, a decoding unit 22 and an image recognition unit 42. The model learning device may include a first identification unit 32, a second identification unit 34, a third image feature extraction unit 38, a fourth image feature extraction unit 39, and a model learning unit 36. The model learning process performed by the model learning unit 36 may be performed in parallel with the data compression / decompression process performed in cooperation between the edge device and the server device (online processing) or at a different time (offline processing). To realize online processing, the data center may be provided with a parameter notification unit (not shown) that transmits the update amounts (described below) of the parameter sets of the first machine learning model, the second machine learning model, the third machine learning model, and the fourth machine learning model determined by the model learning unit 36 to the first identification unit 32, the second identification unit 34, the third image feature extraction unit 38, and the third image feature extraction unit 38, respectively, for each update step. Alternatively, instead of or in addition to the data center, the edge device may further include a first identification unit 32, a second identification unit 34, a third image feature extraction unit 38, a fourth image feature extraction unit 39, and a model learning unit 36. With this configuration, online processing may be realized. To realize online processing, the edge device may include the parameter notification unit described above.

[0018] The input processing unit 14 acquires image data. The input processing unit 14 receives image data from, for example, an imaging unit. The input processing unit 14 may also receive image data from another device. The input processing unit 14 includes, for example, an input interface. The input processing unit 14 may also include an imaging unit. The input processing unit 14 outputs the acquired image data to the encoding unit 12, the first identification unit 32, and the third image feature extraction unit 38. In this application, the image shown in the image data acquired by the input processing unit 14 is sometimes referred to as an "original image," and the image data showing the original image is sometimes referred to as "current image data."

[0019] The encoding unit 12 includes a compression unit 124. The compression unit 124 extracts image features that represent the characteristics of the image shown in the image data input from the input processing unit 14. The data volume of the extracted image features is smaller than that of the image data. The extracted image features may be different from the first to fourth image features described below. The encoding unit 12 uses a second machine learning model when extracting the image features from the image data. The compression unit 124 quantizes the determined image features and generates a data series consisting of one or more quantized values obtained by quantization as compressed data. The compression unit 124 outputs the generated compressed data to the decoding unit 22 and the model learning unit 36.

[0020] The decoding unit 22 includes a restoration unit 224 . The restoration unit 224 de-quantizes a data sequence forming compressed data input from the encoding unit 12 and restores one or more quantized values of image features represented by the de-quantized data sequence. The restoration unit 224 restores an image having characteristics represented by a determined quantization value of one or more as a reconstructed image. The restoration unit 224 uses a third machine learning model when restoring the reconstructed image from the quantization value of one or more. The restoration unit 224 generates reconstructed image data representing the reconstructed image and outputs the generated reconstructed image data to the second identification unit 34 and the fourth image feature extraction unit 39. The compression processing unit 30 includes the compression unit 124 and the restoration unit 224, and functions as a generator that generates restored image data based on image data representing the original image input from the input processing unit 14.

[0021] The first discrimination unit 32 receives image data from the input processing unit 14 and receives third image features from the third image feature extraction unit 38. Using a first machine learning model and with the input third image features as conditions, the first discrimination unit 32 determines a conditional confidence of a first image feature, which is a predetermined image feature, in a specific region, which is a part of the image shown in the input image data. The specific region is a region that is a region of interest (RoI) in which the observer is interested, or a region that is likely to be such. The specific region may be the entire image or a partial region. The first discrimination unit 32 functions as a discriminator for discriminating the first image feature from the image data. The first discrimination unit 32 outputs the determined conditional confidence of the first image feature to the model learning unit 36.

[0022] The second classification unit 34 receives the restored image data from the restoration unit 224 and the third image feature from the third image feature extraction unit 38. Using a fourth machine learning model, the second classification unit 34 determines, based on the input third image feature as a condition, a conditional reliability of a second image feature, which is a predetermined image feature in a feature region, which is a part of the restored image shown in the input restored image data. The second image feature is an image feature of the same type as the first image feature. Therefore, the fourth machine learning model applies the same type of method as the first machine learning model, and uses the same model parameters as the first machine learning model. The second classification unit 34 outputs the determined conditional reliability of the second image feature to the model learning unit 36.

[0023] The second classification unit 34 functions as a classifier for classifying second image features from the restored image data. A parameter set common to the first machine learning model is set in the second classification unit 34 as a parameter set for the fourth machine learning model. If the restored image is completely identical to the original image shown in the image data provided to the first classification unit 32 from the input processing unit 14, the reliability determined by the second classification unit 34 will be equal to the reliability determined by the first classification unit 32. The more the image features of the restored image differ from the image features of the original image, the greater the difference in reliability tends to be.

[0024] The third image feature extraction unit 38 extracts image features for recognizing a subject as third image features from an image represented by image data input from the input processing unit 14. The third image feature is an image feature amount mainly used to recognize the type and state of a subject in image recognition processing. The third image feature is derived separately from the first image feature and the second image feature. The third image feature extraction unit 38 may calculate the third image feature by, for example, performing a predetermined arithmetic process. The third image feature may be a known image feature amount as long as it is an image feature amount useful for recognizing a subject. Examples of known image features that may be used include SIFT (Scaled Invariance Feature Transform) and HoG (Histograms of Oriented Gradients). The third image feature extraction unit 38 may also extract the third image feature from an original image using a fifth machine learning model that is separate from the first to fourth machine learning models. The third image feature extraction unit 38 outputs the extracted third image feature to the first classification unit 32, the second classification unit 34, and the model learning unit 36.

[0025] The fourth image feature extraction unit 39 extracts, as the fourth image feature, an image feature for recognizing the subject from the restored image shown in the restored image data input from the decoding unit 22. The fourth image feature may be an image feature of the same type as the third image feature. If the restored image is completely identical to the original image, the fourth image feature will be equal to the third image feature. The fourth image feature extraction unit 39 may extract the fourth image feature from the restored image using a sixth machine learning model. In this case, the sixth machine learning model is a mathematical model of the same type as the fifth machine learning model, and the same parameter set as the parameter set of the fifth machine learning model is used. The fourth image feature extraction unit 39 outputs the extracted fourth image feature to the model learning unit .

[0026] The model learning unit 36 includes a data amount calculation unit 362 , a feature loss calculation unit 364 , and a parameter update unit 366 . The data amount calculation unit 362 calculates the data amount of the code generated by performing entropy coding on the compressed data input from the compression unit 124. The data amount calculation unit 362 outputs the calculated data amount to the parameter update unit 366.

[0027] The feature loss calculation unit 364 receives the third image feature from the third image feature extraction unit 38 and the fourth image feature from the fourth image feature extraction unit 39. The feature loss calculation unit 364 calculates a feature loss function that indicates the degree of variation from the input third image feature to the input fourth image feature. The feature loss calculation unit 364 outputs the calculated feature loss function to the parameter update unit 366.

[0028] The parameter update unit 366 receives the conditional reliability of the first image feature conditioned on the third image feature from the first identification unit 32, and receives the conditional reliability of the second image feature conditioned on the third image feature from the second identification unit 34. As illustrated in FIG. 4 , the parameter update unit 366 updates the parameter set of the first machine learning model so as to increase (maximize) a first loss function indicating the degree of change from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature. The parameter update unit 366 determines the parameter set of the fourth machine learning model to be equal to the parameter set of the first machine learning model.

[0029] The parameter update unit 366 sequentially calculates update amounts for the parameter set of the first machine learning model for each update step using, for example, a gradient method, and outputs the calculated update amounts to the first identification unit 32 and the second identification unit 34. Gradient methods include steepest descent and stochastic gradient descent, and any of these methods may be used. The first identification unit 32 updates the parameter set of a new first machine learning model by adding the update amount input from the parameter update unit 366 to the parameter set of the first machine learning model currently set. The second identification unit 34 updates the parameter set of a new fourth machine learning model by adding the update amount input from the parameter update unit 366 to the parameter set of a fourth machine learning model currently set. By setting the initial value of the parameter set of the first machine learning model to a value equal to the initial value of the parameter set of the fourth machine learning model, the parameter set of the first machine learning model and the parameter set of the fourth machine learning model become equal. In this application, the process of updating the parameter sets of the first machine learning model and the fourth machine learning model may be referred to as "learning the classifier."

[0030] The parameter update unit 366 receives the conditional confidence from the second identification unit 34 and the feature loss function from the feature loss calculation unit 364 as input. As illustrated in FIG. 5 , the parameter update unit 366 updates the parameter set of the second machine learning model and the parameter set of the third machine learning model so as to minimize (minimize) a second loss function obtained by combining the conditional confidence of the second image feature, which uses the third image feature as a condition, and the feature loss function. The parameter update unit 366 sequentially calculates update amounts for the parameter sets of the second machine learning model and the third machine learning model, for example, using a gradient method, and outputs the calculated update amounts for the parameter set of the second machine learning model to the compression unit 124 and outputs the update amounts for the parameter set of the third machine learning model to the restoration unit 224. The compression unit 124 updates the parameter set of the second machine learning model set currently set by adding the update amounts from the parameter update unit 366 to the sum obtained, as a new parameter set of the second machine learning model. The restoration unit 224 updates the parameter set of the third machine learning model set at that time to the sum obtained by adding the update amount from the parameter update unit 366 as the parameter set of the new third machine learning model.

[0031] The parameter update unit 366 may update the parameter sets of the second machine learning model and the third machine learning model so as to reduce the second loss function obtained by further combining the second loss function with an information loss function based on the amount of data input from the data amount calculation unit 362. In this application, the process of updating the parameter sets of the second machine learning model and the third machine learning model may be referred to as "learning of the generator."

[0032] Furthermore, when the third image feature extraction unit 38 extracts the third image features using a fifth machine learning model and the fourth image feature extraction unit 39 extracts the fourth image features using a sixth machine learning model, the parameter update unit 366 may further update the parameter set of the fifth machine learning model so that the second loss function described above becomes smaller during generator training. The parameter update unit 366 determines the parameter set of the sixth machine learning model so that it is equal to the parameter set of the fifth machine learning model. The parameter update unit 366 further sequentially calculates update amounts for the parameter set of the fifth machine learning model, for example, using a gradient method, and outputs the calculated update amounts for the parameter set of the fifth machine learning model to the third image feature extraction unit 38 and the fourth image feature extraction unit 39. The third image feature extraction unit 38 updates the parameter set of the fifth machine learning model to a new parameter set of the fifth machine learning model by adding the update amount input from the parameter update unit 366 to the parameter set of the fifth machine learning model currently set. The fourth image feature extraction unit 39 updates the parameter set of a new sixth machine learning model to the sum obtained by adding the update amount input from the parameter update unit 366 to the parameter set of the sixth machine learning model that is currently set. By presetting a value equal to the initial value of the parameter set of the fifth machine learning model as the initial value of the parameter set of the sixth machine learning model, the parameter set of the sixth machine learning model becomes equal to the parameter set of the fifth machine learning model.

[0033] In the present application, maximizing the first loss function includes searching for a parameter set that increases the first loss function, and is not limited to absolutely maximizing the first loss function. The first loss function may temporarily decrease during classifier training. Minimizing the second loss function includes searching for a parameter set that decreases the second loss function, and is not limited to absolutely minimizing the second loss function. The second loss function may temporarily decrease during generator training.

[0034] The parameter update unit 366 may alternately repeat the training of the classifier and the training of the generator for each update step of the parameter set. The parameter update unit 366 determines the parameter set of the fourth machine learning model to be equal to the parameter set of the first machine learning model for each update step. Furthermore, when determining the parameter set of the fifth machine learning model, the parameter update unit 366 determines the parameter set of the sixth machine learning model to be equal to the parameter set of the fifth machine learning model for each update step.

[0035] The parameter update unit 366 may repeat the learning of the classifier and the learning of the generator a predetermined number of times, or may perform the learning until it is determined that one of the parameter sets has converged. The parameter update unit 366 can determine whether the first parameter set, and therefore the fourth parameter set, has converged, for example, based on whether the magnitude of the difference between the first loss function before the parameter set update and the first loss function before the update is equal to or less than a predetermined threshold for the magnitude of the difference between the first loss function. Furthermore, the parameter update unit 366 can determine whether the second parameter set and the third parameter set (and the fifth parameter set, if applicable) have converged based on whether the magnitude of the difference between the second loss function before the parameter set update and the second loss function before the update is equal to or less than a predetermined threshold for the magnitude of the difference between the second loss function.

[0036] In training the classifier, the parameter update unit 366 may set a target value of conditional reliability to 1 for an original image in which the first image feature appears, provided that the third image feature appears in the feature region; a target value of reliability to 0 for an original image in which the first image feature or the third image feature does not appear in the feature region; and a target value of reliability to 0 for other image features not appearing in the original image. The parameter update unit 366 may train the classifier so that an estimated value of conditional reliability for the second image feature estimated for a restored image corresponding to an original image in which the first image feature appears, provided that the third image feature appears, and an estimated value of conditional reliability for the second image feature estimated for a restored image corresponding to an original image in which the third image feature or the first image feature does not appear, approach their respective target values. As a result, the ranges of the conditional reliability calculated by the first classifier 32 and the conditional reliability calculated by the second classifier 34 are bounded to real values between 0 and 1. Conversely, the parameter update unit 366 may perform generator training without constraining the estimated values to the respective target values.

[0037] The first identification unit 32 receives an input of the update amount of the parameter set of the first machine learning model from the parameter update unit 366. The second identification unit 34 receives an input of the update amount of the parameter set of the fourth machine learning model (equal to the update amount of the parameter set of the first machine learning model) from the parameter update unit 366. The first identification unit 32 updates the parameter set of the first machine learning model by adding the input update amount of the parameter set of the first machine learning model to the parameter set of the first machine learning model at that time. The second identification unit 34 updates the parameter set of the fourth machine learning model by adding the input update amount of the parameter set of the fourth machine learning model to the parameter set of the fourth machine learning model at that time. The compression unit 124 receives an input of the update amount of the parameter set of the second machine learning model from the parameter update unit 366. The restoration unit 224 receives an input of the update amount of the parameter set of the third machine learning model from the parameter update unit 366. The compression unit 124 and the restoration unit 224 update the parameter set of the third machine learning model at that time by adding the input update amount of the parameter set of the fourth machine learning model to the parameter set of the third machine learning model at that time.

[0038] As described above, in the training of the classifier, the first loss function is maximized. The first loss function indicates the degree of variation between the conditional reliability of the first image feature input from the first classification unit 32 and the conditional reliability of the second image feature input from the second classification unit 34. The conditional reliability of the first image feature and the conditional reliability of the second image feature are each conditioned on the third image feature. The first loss function is an index that quantitatively indicates the change in the reliability of the image features identified by the first classifier 32 and the second classifier 34 due to compression and restoration. The first loss function is also called GAN (Generative Adversarial Network) loss. The first loss function L D For example, as shown in equation (1), quantifies the degree of variation (divergence) between the distribution of the conditional reliability D(x|f) of the first image feature given the third image feature f as a condition and the distribution of the conditional reliability D(G(E(x))|f) of the second image feature given the third image feature f as a condition.

[0039]

number

[0040] In formula (1), E x~p(x) [...] indicates the expected value of.... x indicates the original image. p(x) indicates the probability distribution of the original image x. In other words, x~p(x) indicates the set of data from which the original image x is obtained with the probability distribution p(x), that is, the training data (supervised data) used for learning. Generally, training data is composed of a large amount of image data. E(x) indicates the probability of the code E(x) obtained by encoding the image x. G(E(x)) indicates the restored image x' obtained by decoding the code E(x). Equation (1) shows that the expected value of the sum of the logarithm of the distribution of the conditional reliability D(x|f) of the first image feature and the logarithm of the conditional reciprocal reliability log(1-D(G(E(x)))) of the second image feature, which is conditioned on the third image feature f, is the first loss function L. D The conditional reciprocal confidence of the second image feature corresponds to the difference 1-D(G(E(x))) of the conditional confidence D(G(E(x))|f) of the second image feature, which is conditional on the third image feature f from 1. In equation (1), the conditional confidence D(x|f) of the first image feature and the conditional confidence D(G((x))|f) of the second image feature are in a complementary relationship. That is, an increase in the conditional confidence D(x|f) of the first image feature decreases the first loss function L D decreases, whereas the conditional confidence D(G(E(x))|f) of the second image feature increases, resulting in the first loss function L D In the following description, a function that determines the third image feature f from the original image x is written as F(x). The first image feature and the second image feature are not limited to one type each, and may each include a plurality of types of image features as elements and be configured by combining these elements.

[0041] Furthermore, in the training of the generator, a second loss function is minimized. The second loss function is an index indicating the degree of variation of the restored image x' from the original image x. The second loss function includes a generator loss and a characteristic loss (characteristic loss function) as components. The generator loss indicates the degree of variation of the restored image due to encoding and decoding. In this embodiment, the logarithm of the conditional reliability D(x'|f) of the second image feature conditional on the third image feature f is used as the generator loss. The characteristic loss indicates the degree of variation from the third image feature f to the fourth image feature F(x') due to encoding and decoding. In this embodiment, the L1 norm ||F(x')-F(x)||1 of the difference from the fourth image feature to the third image feature is used as the characteristic loss. The L1 norm is also called the first-order norm. The L1 norm is equivalent to the sum of the absolute values of the element values of a vector, and is a scalar quantity that takes a smaller value as the vector elements become sparser. Using the L1 norm guides updates to individual element values without excessive computational overhead.

[0042] The second loss function may further include a bitrate loss as a component. In this application, the bitrate loss is also referred to as an "information loss function." The bitrate loss indicates the amount of compressed data for the original image x. The compressed data includes codes obtained by compressing and encoding the original image x. The amount of data input from the data amount calculation unit 362 is used as the bitrate loss.

[0043] In the example of equation (2), the second loss function L E,G,Q is given as the expected value of the weighted sum of the generator loss, feature loss, and bitrate loss under the occurrence probability p(x) of the current image x. The generator loss, feature loss, and bitrate loss are shown in the first, second, and third terms on the right-hand side of equation (2), respectively. α and β are weighting coefficients for the generator loss and feature loss, respectively. The weighting coefficients α and β are positive real values. The weighting coefficient for bitrate loss is normalized as 1.

[0044]

number

[0045] The first to sixth machine learning models may be any type of neural network, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). The first to sixth machine learning models may be mathematical models other than neural networks, such as a random forest. However, the fourth machine learning model uses the same type of mathematical model as the first machine learning model. The sixth machine learning model uses the same type of mathematical model as the fifth machine learning model.

[0046] Next, we will explain an example configuration of the compression unit 124. Fig. 2 is a schematic block diagram showing an example configuration of the compression unit 124. The compression unit 124 is configured to include a characteristic analysis unit 1242, a first distribution estimation unit 1244, and a first sampling unit 1246.

[0047] The characteristic analysis unit 1242 analyzes an image feature quantity that indicates the characteristics of an image represented by input image data using a first type machine learning model as a first characteristic value, and outputs the determined first characteristic value to the first distribution estimation unit 1244. The image data typically indicates a signal value for each pixel. The first type machine learning model is a mathematical model that constitutes a part of the second machine learning model. The image feature quantity to be analyzed may be a specific image feature quantity, such as a brightness gradient or edge distribution. If the first type machine learning model is a neural network, the output value for each node included in a predetermined layer of the layer may be used. The predetermined layer is not limited to an output layer, but may also be an intermediate layer.

[0048] The first distribution estimation unit 1244 uses each element value included in the first characteristic value input from the characteristic analysis unit 1242 as an input value, and estimates a first probability distribution of quantized values for each input value using a second type machine learning model. The first distribution estimation unit 1244 outputs the estimated first probability distribution to the first sampling unit 1246. The quantized values are distributed within a predetermined value range and can be discretized numerical values. The second type machine learning model constitutes a part of the second machine learning model and is a mathematical model separate from the first type machine learning model. The first probability distribution is configured to include a probability for each quantized value within a predetermined value range.

[0049] The second type machine learning model is, for example, a mixture model that defines, for each quantized value, a probability distribution that includes a normalized probability of the product of the prior probability of that quantized value and the conditional probability of the input value given that quantized value as the first probability distribution. Normalization is achieved by dividing by the sum of the products of each quantized value within the range.

[0050] The first distribution estimation unit 1244 calculates the conditional probability of an input value for each quantized value and the prior probability for each quantized value using, for example, a Gaussian Mixture Model (GMM). The Gaussian Mixture Model is a mathematical model that uses a predetermined number of normal distributions (Gaussian functions) as basis functions and expresses a continuous probability distribution as a linear combination of these basis functions. Therefore, the parameter set of the second type machine learning model includes weights, mean values, and variances, which are parameters of each normal distribution. All of these parameters are expressed as real numbers. Therefore, the conditional probabilities, prior probabilities, and the probabilities for each quantized value determined using these are differentiable with respect to the above parameters.

[0051] The first sampling unit 1246 samples one quantized value from a set range of values according to the first probability distribution input from the first distribution estimation unit 1244, and determines the sampled quantized value as a first sample value. The first sampling unit 1246 generates, for example, a pseudo-random number that becomes one of the quantized values within the range of values, so that the pseudo-random number appears with the probability of that quantized value. The first sampling unit 1246 determines the generated pseudo-random number as a first sample value. The first sampling unit 1246 accumulates the determined first sample values in the order in which they were obtained, and generates a data sequence including a predetermined number of first sample values as compressed data. The first sampling unit 1246 outputs the generated compressed data to the decoding unit 22.

[0052] Next, we will explain an example configuration of the restoration unit 224. Fig. 3 is a schematic block diagram showing an example configuration of the restoration unit 224 according to this embodiment. The restoration unit 224 is configured to include a second distribution estimation unit 2242, a second sampling unit 2244, and a data generation unit 2246.

[0053] The second distribution estimation unit 2242 estimates, as a second probability distribution, a probability distribution corresponding to each of the first sample values included in the data sequence forming the compressed data input from the encoding unit 12, using a third type machine learning model. The second distribution estimation unit 2242 outputs second probability distribution information indicating the estimated second probability distribution to the second sampling unit 2244. The third type machine learning model may be any mathematical model that can determine a probability distribution using a continuous probability density function corresponding to the first sample values. For example, a GMM can be used as the third type machine learning model. In this case, the second probability distribution information includes weighting coefficients, mean values, and variances, which are parameters of each normal distribution.

[0054] The second sampling unit 2244 samples one real value from a set range of values according to the second probability distribution given by the second probability distribution information input from the second distribution estimation unit 2242. Here, the second sampling unit 2244 generates, for example, a pseudo-random number that becomes a real value within the range of values, so that the pseudo-random number appears with a probability corresponding to the real value, and determines a real value by sampling the generated pseudo-random number. The second sampling unit 2244 then quantizes the sampled real value and determines the quantized value obtained as the second sample value. The second sampling unit 2244 outputs the determined second sample value to the data generation unit 2246.

[0055] The data generation unit 2246 uses the second sample values input from the second sampling unit 2244 as element values and determines a second characteristic value including one or more element values. For the image feature having the determined second characteristic value, the data generation unit 2246 generates restored image data of a restored image having characteristics represented by the image feature using a fourth type machine learning model. The data generation unit 2246 outputs the generated restored image data to the fourth image feature extraction unit 39 and the second identification unit 34. The fourth type machine learning model constitutes a part of the third machine learning model and is a machine learning model separate from the third type machine learning model. The fourth type machine learning model may be, for example, a mathematical model of the same type as the first type machine learning model. If the first type machine learning model is a neural network, the fourth type machine learning model may also be a neural network. According to the configurations shown in FIGS. 2 and 3, the image feature of each original image is non-deterministically quantized.

[0056] 2 and 3. In the compression unit 124, the first distribution estimation unit 1244 and the first sampling unit 1246 may be omitted. In that case, the compression unit 124 may determine a quantization value using a predetermined quantization interval for the first characteristic value obtained from the characteristic analysis unit 1242. The compression unit 124 outputs a data sequence obtained by accumulating the determined quantization values as first sample values to the decoding unit 22 as compressed data. The second distribution estimation unit 2242 and the second sampling unit 2244 may be omitted from the restoration unit 224. In this case, the restoration unit 224 outputs the first sample value included in the data sequence forming the compressed data input from the encoding unit 12 to the data generation unit 2246 as the second sample value.

[0057] Next, an example of image compression and decompression processing according to this embodiment will be described below with reference to a flowchart shown in FIG. (Step S102) The input processing unit 14 acquires image data to be processed and outputs it to the compression unit 124. (Step S104) The compression unit 124 compresses the data amount of the image data using the second machine learning model, and generates compressed data consisting of a data sequence including codes that indicate the characteristics of the original image. The compression unit 124 outputs the generated compressed data to the decoding unit 22.

[0058] (Step S110) The restoration unit 224 uses the third machine learning model to expand the data amount of the data sequence forming the compressed data input from the encoding unit 12, and restores it to restored image data indicating a restored image. The restoration unit 224 outputs the restored image data to the fourth image feature extraction unit 39. (Step S112) The fourth image feature extraction unit 39 extracts a fourth image feature from the restored image data input from the restoration unit 224. Then, the processing in Fig. 6 ends. The extracted fourth image feature is used, for example, in image recognition processing.

[0059] Next, an example of the model learning process according to this embodiment will be described below. Fig. 7 is a flowchart showing an example of the model learning process according to this embodiment. (Step S202) The third image feature extraction unit 38 extracts third image features from the original image shown in the image data acquired from the input processing unit 14. The third image feature extraction unit 38 outputs the extracted third image features to the first identification unit 32. (Step S204) The first identification unit 32 uses the first machine learning model to calculate a conditional reliability of the first image feature, which is based on the third image feature input from the third image feature extraction unit 38. The first image feature is identified from the original image shown in the image data acquired from the input processing unit 14. (Step S206) The data amount calculation unit 362 determines the data amount of the compressed data acquired from the compression unit 124.

[0060] (Step S208) The second identification unit 34 uses the fourth machine learning model to calculate a conditional reliability of the second image feature, which is based on the third image feature input from the third image feature extraction unit 38. The second image feature is identified from the restored image shown in the restored image data obtained from the restoration unit 224. (Step S210) The parameter update unit 366 calculates the amount of update for the parameter set of the first machine learning model so as to maximize a first loss function indicating the degree of change from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature (learning of the classifier).

[0061] (Step S212) The fourth image feature extraction unit 39 extracts a fourth image feature from the restored image indicated in the restored image data input from the restoration unit 224. The fourth image feature extraction unit 39 outputs the extracted fourth image feature to the parameter update unit 366. (Step S214) The parameter update unit 366 calculates the update amount of the parameter set of the second machine learning model and the update amount of the parameters of the third machine learning model so as to minimize the second loss function obtained by combining the conditional reliability of the second image feature conditional on the third image feature and the feature loss function indicating the degree of variation from the third image feature to the fourth image feature (learning of the generator).

[0062] (Step S216) The parameter update unit 366 updates the parameter sets of the first to fourth machine learning models using the update amounts determined for each. (Step S218) The parameter update unit 366 determines whether the parameter set has converged. If it is determined that it has converged (step S218 YES), the processing in Fig. 7 ends. If it is determined that it has not converged (step S218 NO), the processing returns to step S202.

[0063] 7 may be executed in parallel with the process shown in FIG. 6 (online learning) or independently of the process shown in FIG. 6 (offline learning). The model learning unit 36 may include functional units corresponding to the input processing unit 14, the compression unit 124, the restoration unit 224, the first classification unit 32, the second classification unit 34, and the fourth image feature extraction unit so that the model learning unit 36 can execute the model learning process independently. The information processing system 1 may be realized as an information processing device including the model learning unit 36.

[0064] Next, an application example of the information processing system 1 will be described. FIG. 8 is a schematic block diagram showing an application example of the information processing system 1a according to this embodiment. The information processing system 1a is an application example to a remote monitoring system. The monitored object is, for example, traffic conditions on a road. The information processing system 1a further comprises an imaging unit 16 and a monitoring support device 40 in addition to the information processing system 1. The monitoring support device 40 comprises a decoding unit 22, an image recognition unit 42, a detection unit 44, a display processing unit 46, a display unit 47, and an operation input unit 48.

[0065] The image capturing unit 16 captures an image within a predetermined field of view and outputs image data representing the captured image to the input processing unit 14. The monitored area is included in the field of view. The image capturing unit 16 is, for example, a digital video camera. In the example of FIG. 8, the input processing unit 14 is configured separately from the image capturing unit 16.

[0066] The image recognition unit 42 includes a fourth image feature extraction unit 39. The image recognition unit 42 performs image recognition processing using a known method with the fourth image feature extracted by the fourth image feature extraction unit 39 to generate recognition information indicating the recognition result. The recognition result includes, for example, the type of subject (e.g., vehicle or pedestrian), the state of the subject (e.g., moving speed or direction), and other objects and their display positions. In the image recognition processing, a machine learning model separate from the first to sixth machine learning models may be used, or a machine learning model partially including the sixth machine learning model used to extract the fourth image feature may be used. The image recognition unit 42 outputs the generated recognition information to the detection unit 44. The image recognition unit 42 outputs the restored image data input from the decoding unit 22 to the display processing unit 46.

[0067] The detection unit 44 detects recognition information indicating a predetermined event (e.g., the approach of a vehicle to another object (e.g., a vehicle or a pedestrian), or a traffic jam on a road) to be notified to a user (e.g., a monitor) using a predetermined detection rule set in advance from the recognition information input from the image recognition unit 42 (event detection). The detection unit 44 may reject recognition information indicating other events. The detection unit 44 outputs the detected recognition information to the display processing unit 46.

[0068] The display unit 47 displays a display screen based on the display screen data input from the display processing unit 46. The display unit 47 is, for example, a display. The operation input unit 48 receives a user operation and outputs operation information corresponding to the received operation to the display processing unit 46. The operation input unit 48 may be configured to include dedicated members such as buttons and knobs, or may be configured to include general-purpose members such as a touch sensor, a mouse, and a keyboard.

[0069] The display processing unit 46 constitutes a user interface together with the display unit 47 and the operation input unit 48. The display processing unit 46 constructs a display screen in which part or all of the restored image shown in the restored image data input mainly from the image recognition unit 42 is arranged in a predetermined display area, and performs processing for displaying the display screen on the display unit 47.

[0070] The display processing unit 46 controls the display function of the display screen in accordance with operation information input from the operation input unit 48. Display screen data showing a display screen including a restored image is output to the display unit 47. The display unit 47 displays the display screen shown in the display screen data input from the display processing unit 46. The display processing unit 46 updates the feature region based on, for example, area instruction information related to the display region of the restored image input from the operation input unit 48. The updated feature region can be set in accordance with an operation by a user who views the restored image included on the display screen. The update processing unit 462 acquires, as the area instruction information, an area that is a part of the original image or the restored image and is indicated by the operation information from the operation input unit as a new feature region.

[0071] The update processing unit 462 may have a dedicated function for explicitly identifying a feature region from a restored image in response to operation information, or may have a function for implicitly identifying a feature region. When implicitly identifying a feature region, the update processing unit 462 may estimate an area corresponding to the display frame of the display screen as a feature region when an operation that suggests a user's interest in a specific region is performed using a function for adjusting the display size or display position of the restored image, and no instruction to change the display size or display position is given for a predetermined waiting time (e.g., 1-3 seconds) or longer. The operation that suggests a user's interest is, for example, a change in display position, enlargement, or a combination thereof. The update processing unit 462 outputs feature region information indicating information about a new feature region to the parameter update unit 366. The output feature region information may be used for training a classifier.

[0072] The update processing unit 462 may further output the recognition information acquired from the image recognition unit 42 to the display unit 47 and acquire object information regarding the characteristics of the object in the feature region. Here, the characteristics of the object may be set in accordance with an operation by a user who views the restored image. The update processing unit 462 acquires object information indicating the characteristics of the object in the feature region from operation information input from the operation input unit. The update processing unit 462 outputs the acquired object information to the parameter update unit 366. The output feature region information may be used in training the generator for a fourth image feature used to detect the object, and ultimately for training the third image feature. In this case, the parameter update unit 366 may set, as correct answer information, a target value of the conditional reliability for a known first image feature included in the original image to 1, and a target value of the conditional reliability for other image features not included in the original image to 0 in the updated feature region of the original image. The parameter update unit 366 may update the parameter sets of the individual machine learning models so that the estimated reliability of the second image feature estimated for the restored image and the estimated reliability of the other image features approach their respective target values.

[0073] As described above, according to the information processing system 1 of this embodiment, a first machine learning model is used on the original image to identify a first image feature in a feature region of the original image, a second machine learning model is used on the original image to generate compressed data with a reduced data volume, a restored image of the original image is generated from the compressed data using a third machine learning model, and a fourth machine learning model is used on the restored image to identify a second image feature in a feature region of the restored image. In addition, the information processing system 1 extracts a third image feature for recognizing the subject from the original image and extracts a fourth image feature for recognizing the subject from the restored image, shares a parameter set of the fourth machine learning model with the parameter set of the first machine learning model, determines the parameter set of the first machine learning model so that a first loss function indicating the degree of variation from the reliability of the first image feature conditioned on the third image feature to the reliability of the second image feature conditioned on the third image feature is larger, and determines the parameter sets of the second machine learning model and the third machine learning model so that a second loss function obtained by combining the reliability of the second image feature conditioned on the third image feature and a feature loss function indicating the degree of variation from the third image feature to the fourth image feature is smaller.

[0074] According to this configuration, the parameter sets of the first machine learning model to the fourth machine learning model are determined so as to obtain a restored image from which a second image feature for discrimination can be extracted so that the variation from the first image feature for discrimination is significant, using the third image feature for recognition extracted from the original image and used for image recognition as a condition. Therefore, the restored image obtained using the second machine learning model and the third machine learning model has the second image feature that is a condition for the third image feature and that varies significantly from the first image feature, thereby improving visual quality. Furthermore, the fourth image feature can be extracted from the restored image using the same method as the method for extracting the third image feature from the original image so that the variation from the third image feature is minimal. Therefore, it is possible to achieve both the subjective quality visually perceived from the restored image and the recognition rate of image recognition using the fourth image feature extracted from the restored image.

[0075] Fourth image features extracted from restored images obtained without using the third image features used for image recognition as conditions tend to differ significantly from the ideal third image features. Figure 12 illustrates the distribution of fourth image features for each vehicle type to be recognized (shaded) and the distribution of third image features for recognizing route buses (solid). The horizontal and vertical axes represent the height and window size of the recognized vehicle as element values of the third and fourth image features, respectively. In this example, the range of the fourth image feature that should be recognized as a "route bus" is eroded into the range recognized as a "minivan" or "large truck," and the range of the fourth image feature that should be recognized as a "tourist bus" is eroded into the range recognized as a "route bus." In contrast, in this embodiment, by using restored images obtained using a parameter set conditioned on the third image features, variation from the third image features to the fourth image features is suppressed. This ensures accurate image recognition.

[0076] Furthermore, the second loss function may be further combined with an information loss function based on the information amount of the compressed data. According to this configuration, it is possible to reduce the amount of compressed data involved in transmitting the original image while simultaneously improving the visual quality of the restored image and the recognition rate of the image. FIG. 13 illustrates the relationship between the recognition rate and bit rate obtained by performing image recognition processing on restored images obtained using this embodiment and other methods. Generally, the higher the bit rate, the higher the recognition rate, but this is higher than when using restored images obtained using other methods. When using restored images obtained using Method A, a recognition rate almost equivalent to that of this embodiment was obtained. In Method A, restored images were generated using model parameters obtained without classifier training in this embodiment. As a result, the subjective quality of the restored images tends to be inferior. Recognition rates for restored images generated using other methods are significantly lower than that of this embodiment. In Method B, restored images were generated using a parameter set determined without conditioning on the third image feature in this embodiment. This method also results in degraded subjective quality. Note that Methods C and D are both methods proposed by Balle et al. (2018). Method E refers to a video encoding / decoding method specified in ITU-T H.264. Method F refers to a video encoding / decoding method specified in ITU-T H.265. Method G refers to the method proposed by Mentzer et al. (2020). Method I refers to the JPEG (Joint Photographic Experts Group) method.

[0077] FIG. 14 shows (a) an original image, (b) a restored image according to this embodiment, and (c) a comparative example. The comparative example is a restored image using a parameter set obtained by unconditional learning using the third image feature. In the illustrated example, the restored image according to this embodiment has higher subjective quality than the restored image according to the comparative example. Unlike the comparative example, the restored image according to this embodiment does not exhibit block noise, and even distant scenes are clearly reproduced.

[0078] FIG. 15 shows (a) an original image, (b) a restored image according to this embodiment, and (c) a comparative example. The comparative example shows a restored image using HEVC (High Efficiency Video Coding). Compression and restoration were performed so that the bit rates between (b) and (c) were equal. In this example, too, the restored image according to this embodiment has higher subjective quality than the restored image according to the comparative example. The restored image according to this embodiment is clearly reproduced without the blurred haze, stripes, and other noise that appear in the comparative example.

[0079] Next, other embodiments will be described. The following description will mainly focus on the differences from the first embodiment. Unless otherwise specified, common reference numerals will be used for configurations and processes common to the first embodiment, and the description thereof will be used. Common reference numerals may also include cases where the parent number (e.g., "1" in "information processing system 1a"), which is part of the reference numeral, is common, but the child number (e.g., "a") is different.

[0080] Second Embodiment Next, a second embodiment will be described. The third image feature according to the second embodiment includes, as elements, image features for recognizing a plurality of types of subjects. Consequently, the fourth image feature also includes image features for recognizing the same types of subjects as the plurality of types. Image recognition processing using the fourth image feature can improve the recognition accuracy of the type of subject corresponding to the image features included as elements. An information processing system 1b (not shown) according to this embodiment includes a third image feature extraction unit 38b instead of the third image feature extraction unit 38. Fig. 9 is a schematic block diagram showing an example of the functional configuration of the third image feature extraction unit 38b according to this embodiment.

[0081] 9, the third image feature extraction unit 38b extracts three types of image features for recognition, connects the extracted three types of image features, and outputs the connected result as a third image feature. The third image feature extraction unit 38b includes a first type image feature extraction unit 382-1 to a third type image feature extraction unit 382-3, and a connection unit 384. The first type image feature extraction unit 382-1 to the third type image feature extraction unit 382-3 each include a mathematical model for calculating the first type image feature to the third type image feature from the original image, and output the calculated first type image feature to the connection unit 384. The concatenation unit 384 concatenates in parallel the calculated first to third image features input from the first to third image feature extraction units 382-1 to 382-3, respectively, to configure a third image feature. The concatenation unit 384 outputs the configured third image feature to the first identification unit 32 and the second identification unit 34.

[0082] The fourth image feature extraction unit 39 has a configuration similar to that of the third image feature extraction unit 38. That is, the fourth image feature extraction unit 39 extracts multiple types of image features from the restored image and configures the fourth image feature by linking the multiple types of extracted image features. The description of the third image feature extraction unit 38 will be cited for the function and configuration of the fourth image feature extraction unit 39. The number of image features included as elements in the third type image feature and the fourth type image feature is not limited to three, but may be two, four or more.

[0083] Individual image features that are elements of the third image feature and the fourth image feature may be used for conditioning in the first classification unit 32 and the second classification unit 34, respectively, and the obtained reliability or intermediate value may be linked between types of image features. Each individual image feature is represented by a vector having multiple element values, and the number of dimensions (number of elements) may differ between types of image features. The first classification unit 32 and the second classification unit 34 may resample each individual image feature that is the element so that the number of dimensions is equal to the number of dimensions of the first image feature and the second image feature for each type of image feature. In resampling, downsampling is performed to reduce the number of dimensions of the image feature so that it is equal to the number of dimensions of the first image feature or the second image feature, and oversampling is performed to increase the number of dimensions of the image feature. Known interpolation processes may be applied to downsampling or oversampling.

[0084] 10 is a schematic block diagram showing an example of the configuration of the first identification unit 32b according to this embodiment. Here, the first identification unit 32b as a whole forms a CNN, and three types of image features are included as elements of the third image feature. The first identification unit 32b includes a first image feature extraction unit 321, resampling units 322-1 through 322-3, concatenation units 324-1 through 324-3, convolution processing units 325-1 through 325-3, pooling units 326-1 through 326-3, concatenation unit 327, and normalization unit 328.

[0085] The first image feature extraction unit 321 extracts first image features from the original image using a predetermined first image feature extraction model. The first image feature extraction unit 321 outputs the extracted first image features to the resampling units 322-1 to 322-3. The first image feature and the first to third image features may be represented, for example, as bitmaps in which color signal values are distributed two-dimensionally for each color. Color signal values across different colors are superimposed in the height direction. The bitmaps each have signal values for sample points arranged at regular intervals in the horizontal and vertical directions on a two-dimensional plane. In this example, it is assumed that the samples of the first image feature and the first to third image features form three-dimensional data distributed horizontally, vertically, and heightwise. The term "three-dimensional" refers to the number of dimensions of the space in which the samples are arranged, and does not refer to the number of elements constituting each image feature, i.e., the number of samples. The number of dimensions involved in resampling is expressed by the number of samples in the horizontal and vertical directions.

[0086] The resampling units 322-1 to 322-3 resample the first image features input from the first image feature extraction unit 321 and convert the number of dimensions for each color so that they are equal to the number of dimensions of the first to third type image features, respectively. The resampling units 322-1 to 322-3 output the converted first image features to the connection units 324-1 to 324-3, respectively.

[0087] The connection unit 324-1 receives the transformed first image feature and the first type image feature from the resampling unit 322-1. The connection unit 324-1 connects the transformed first image feature and the first type image feature by stacking them in the height direction, and outputs the obtained first type connected feature to the convolution processing unit 325-1. The connection unit 324-2 receives the transformed first image features and second image features from the resampling unit 322-2. The connection unit 324-2 connects the transformed first image features and second image features by stacking them in the height direction, and outputs the obtained second connected features to the convolution processing unit 325-2. The connection unit 324-3 receives the converted first image features and third image features from the resampling unit 322-3. The connection unit 324-3 connects the converted first image features and third image features by stacking them in the height direction, and outputs the obtained third connected feature to the convolution processing unit 325-3.

[0088] The convolution processing units 325-1 to 325-3 each use color signal values constituting the first to third type connected features as input values, perform convolution for each input value, and calculate an output value. The number of samples of the calculated output value may be equal to or less than the number of samples of the input value. However, at this stage, it is assumed that each sample is distributed in a three-dimensional space. The convolution processing units 325-1 to 325-3 may each have a configuration similar to that of a CNN. The convolution processing units 325-1 to 325-3 output convolution outputs consisting of output values for each element to the pooling units 326-1 to 326-3.

[0089] Pooling units 326-1 to 326-3 average the input values of individual samples forming the convolution output input from convolution processing units 325-1 to 325-3, respectively, in the horizontal and vertical directions for each two-dimensional plane (global pooling), and output pooling output having the obtained average value as the output value to connection unit 327. The pooling output becomes one-dimensional data (vector) containing multiple output values as elements in the height direction.

[0090] The concatenation unit 327 connects the pooling outputs input from the pooling units 326-1 to 326-3 by combining them in the height direction to form a concatenated output. The concatenation unit 327 outputs the constructed concatenated output to the normalization unit 328. The normalization unit 328 calculates a weighted sum of input values for each sample forming the concatenated output input from the concatenation unit 327, and normalizes the calculated weighted sum so that the range of the weighted sum is equal to or greater than 0 and equal to or less than 1. The normalization unit 328 outputs the calculated value obtained by normalization as a reliability to the parameter update unit 366. The normalization unit 328 is realized using, for example, a multilayer perceptron (MLP). The second discrimination section 34b (not shown) may have a similar configuration to the first discrimination section 32b. The description of the first discrimination section 32b is cited for the function and configuration of the second discrimination section 34b.

[0091] <Third embodiment> Next, a description will be given of a third embodiment. The information processing system 1 according to the third embodiment includes a filter setting unit 365 and a filter processing unit 367. The filter setting unit 365 sets, in the filter processing unit 367, a spatial filter whose spatial frequency characteristics vary depending on the position. The filter processing unit 367 uses the spatial filter set by the filter setting unit 365 to perform filtering on the original image indicated in the image data input from the input processing unit 14. The filter processing unit 367 outputs image data indicating the processed original image (hereinafter, sometimes referred to as the "processed image") to the compression unit 124, the first identification unit 32, and the third image feature extraction unit 38.

[0092] The spatial filter can be a low-pass filter (LPF). The spatial filter may be, for example, a Gaussian filter. A Gaussian filter is a low-pass filter whose filter coefficients are determined based on a normal distribution with the pixel to be processed as its origin. The Gaussian filter has the characteristic of blocking high-frequency components with high spatial frequencies and leaving low-frequency components as the standard deviation or variance (hereinafter collectively referred to as "variance, etc.") of the normal distribution increases. When filtering is performed using such a spatial filter, if the low-pass characteristics of a certain region are high, the processed image will be less clear than the surrounding regions. The spatial filter may be configured as a sharpness map in which spatial frequency characteristics are set for each pixel. The sharpness map can be configured using the distribution of the standard deviation of a Gaussian filter in one frame of an image.

[0093] The sharpness map may define the sharpness distribution using a normal distribution separate from the individual Gaussian filters. For example, the sharpness distribution center, which is the position where sharpness is lowest, is expressed by the coordinates of the origin of the normal distribution representing the sharpness distribution, and the sharpness variance, which indicates the spread of sharpness, is expressed by the variance of the normal distribution. Display regions having low-pass characteristics in the spatial filter may be avoided by not including the feature regions identified by the first identification unit 32. This prevents loss of high-frequency components with high spatial frequencies in the feature regions of the processed image.

[0094] In training the parameter sets of the machine learning models, the filter setting unit 365 may set spatial filters with different spatial frequency characteristics for each frame of the training data. In training the classifier, the parameter updating unit 366 determines the parameter sets of the first machine learning model using a first image feature identified from the processed image, a second image feature identified from a restored image based on the processed image obtained from compressed data generated from the processed image, and a third image feature extracted from the processed image. In training the generator, the parameter updating unit 366 determines the parameter sets of the second machine learning model and the third machine learning model using the first image feature, the second image feature, the third image feature, and a fourth image feature identified from the restored image based on the processed image.

[0095] When setting spatial filters with different spatial frequency characteristics for each frame, the filter setting unit 365 may, for example, randomly determine the sharpness distribution center and sharpness variance, which represent the sharpness distribution, for each frame using pseudorandom numbers. This allows images expressing different patterns due to differences in sharpness distribution to be synthesized, and the synthesized images are used as training data. Even if the amount of training data is limited, the machine learning model can be trained to obtain restored images that enable high-quality and high-precision image recognition.

[0096] <Fourth embodiment> Next, a fourth embodiment will be described. In the learning of the generator, the parameter update unit 366 according to this embodiment uses the larger value (maximum value) of the information amount of compressed data -log(Q(z)) and the target value B of that information amount as the bit rate loss. As shown in equation (3), the bit rate loss max(-log(Q(z)),B) is calculated by the second loss function L E,G,Q In the generator training, the second loss function L E,G,Q is minimized, the parameter sets of the second machine learning model related to the compression unit 124 and the third machine learning model related to the restoration unit 224 are determined so that the amount of information in the compressed data −log(Q(z)) does not exceed the target value B.

[0097]

number

[0098] (Minimum configuration) Next, the minimum configuration of the above embodiment will be described. Fig. 16 is a schematic block diagram showing an example of the minimum configuration of an information processing system 1 of the present application. The information processing system 1 includes a first identification unit 32 that identifies a first image feature in a feature region of the original image using a first machine learning model for the original image, a compression unit 124 that generates compressed data with a reduced data volume using a second machine learning model for the original image, a restoration unit 224 that generates a restored image of the original image from the compressed data using a third machine learning model, a second identification unit 34 that identifies a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image, a third image feature extraction unit 38 that extracts a third image feature for object recognition from the original image, a fourth image feature extraction unit 39 that extracts a fourth image feature for object recognition from the restored image, and a model learning unit 36. The model learning unit 36 shares a parameter set of the fourth machine learning model with the parameter set of the first machine learning model, determines the parameter set of the first machine learning model so that a first loss function indicating the degree of change from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature is larger, and determines the parameter sets of each of the second machine learning model and the third machine learning model so that a second loss function obtained by combining the conditional reliability of the second image feature conditioned on the third image feature and the feature loss function indicating the degree of change from the third image feature to the fourth image feature is smaller.

[0099] 17 is a schematic block diagram showing an example of a minimum configuration of an information processing device 50. The information processing device 50 includes a model learning unit 36 that determines a parameter set for the first machine learning model so that a first loss function indicating a degree of variation from a conditional confidence of a first image feature identified in a feature region of an original image using a first machine learning model for the original image, using a third image feature for recognizing an object extracted from the original image as a condition, to a conditional confidence of a second image feature identified in a feature region of the restored image using a fourth machine learning model having a parameter set common to the first machine learning model for a restored image of the original image generated using a second machine learning model for the original image from compressed data with a reduced data volume generated using a second machine learning model for the original image, the conditional confidence being the third image feature; and determines parameter sets for each of the second and third machine learning models so that a second loss function obtained by combining the conditional confidence of the second image feature based on the third image feature and a feature loss function indicating a degree of variation from the third image feature to a fourth image feature for recognizing an object extracted from the restored image is reduced.

[0100] The above-mentioned devices, such as edge devices, server devices, information processing devices, and monitoring support devices, may each include a computer system. The computer system includes one or more processors, such as a CPU (Central Processing Unit). The above-mentioned processes are stored in computer-readable storage media in the form of programs for each device or apparatus, and the computer reads and executes these programs to perform the processes. The computer system includes software such as an operating system (OS), device drivers, and utility programs, as well as hardware such as peripheral devices. Furthermore, the term "computer-readable storage medium" refers to portable media such as magnetic disks, magneto-optical disks, read-only memories (ROMs), and semiconductor memories, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable storage medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used to transmit programs over networks like the Internet or communication lines like telephone lines, and devices that store programs for a fixed period of time, such as volatile memory within the computer systems that serve as servers or clients in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or it may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system, a so-called differential file (differential program).

[0101] Furthermore, some or all of the devices or apparatuses in the above-described embodiments may be realized as integrated circuits such as LSI (Large Scale Integration). Each functional block of each device or apparatus may be individually implemented as a processor, or some or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0102] The above embodiment may be realized as follows. (Supplementary Note 1) A first identification means for identifying a first image feature in a feature region of an original image using a first machine learning model for the original image; a compression means for generating compressed data with a reduced amount of data using a second machine learning model for the original image; a restoration means for generating a restored image of the original image from the compressed data using a third machine learning model; a second identification means for identifying a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image; a third image feature extraction means for extracting a third image feature for recognizing an object from the original image; a fourth image feature extraction means for extracting a fourth image feature for recognizing the object from the restored image; and a model learning means for determining the parameter sets of the second and third machine learning models so that a first loss function indicating the degree of change from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature becomes larger, and a second loss function obtained by combining the conditional reliability of the second image feature conditioned on the third image feature and a feature loss function indicating the degree of change from the third image feature to the fourth image feature becomes smaller.

[0103] (Supplementary Note 2) In the information processing system of Supplementary Note 1, the second loss function is further combined with an information amount loss function based on the information amount of the compressed data.

[0104] (Supplementary Note 3) In the information processing system of Supplementary Note 2, the information loss function is the maximum value between the information amount of the compressed data and a target value of the information amount.

[0105] (Supplementary Note 4) In the information processing system according to any one of Supplementary Note 1 to Supplementary Note 3, the third image feature and the fourth image feature each include image features for recognizing a plurality of types of subjects.

[0106] (Supplementary Note 5) An information processing system according to any one of Supplementary Note 1 to Supplementary Note 4, comprising a filter processing means that generates a processed image by filtering an original image with spatial frequency characteristics that differ for each frame, and the model learning means determines a parameter set for the first machine learning model using a first image feature identified from the processed image, a second image feature identified from a restored image based on the processed image obtained from compressed data generated from the processed image, and the third image feature extracted from the processed image, and determines parameter sets for each of the second machine learning model and the third machine learning model using the first image feature, the second image feature, the third image feature, and a fourth image feature identified from the restored image based on the processed image.

[0107] (Supplementary Note 6) An information processing system according to any one of Supplementary Note 1 to Supplementary Note 5, comprising: a transmitting device including the first identification means and the compression means; a receiving device including the restoration means and the second identification means; and a parameter notification means, wherein the model learning means includes a first identification means for learning that identifies the first image feature for the original image; a compression means for learning that generates the compressed data for the original image; a restoration means for learning that generates a restored image of the original image from the compressed data; and a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image. The model learning system includes a second identification means for learning to identify features, a third image feature extraction means for extracting third image features for recognizing the subject from the original image, and a fourth image feature extraction means for extracting fourth image features for recognizing the subject from the restored image, and the parameter notification means notifies the first identification means, the compression means, the restoration means, and the second identification means, respectively, of the parameter set of the first machine learning model, the parameter set of the second machine learning model, the parameter set of the third machine learning model, and the parameter set of the fourth machine learning model determined by the model learning means.

[0108] (Supplementary Note 7) In the information processing system of any one of Supplementary Note 1 to Supplementary Note 6, the first image feature, the second image feature, the third image feature, and the fourth image feature each have a plurality of element values, and the first identification means includes a first image feature extraction means for extracting the first image feature from the original image, a first resampling means for resampling the first image feature so that the number of elements of the first image feature becomes equal to the number of elements of the third image feature, and a first resultant combining means for combining the resampled first image feature and the third image feature. and a first reliability calculation means for calculating a conditional reliability of the first image feature from a combined image feature, and the second identification means comprises a second image feature extraction means for extracting the second image feature from the restored image, a second resampling means for resampling the second image feature so that the number of elements of the second image feature is equal to the number of elements of the fourth image feature, and a second reliability calculation means for calculating a conditional reliability of the second image feature from a second combined image feature that combines the resampled second image feature and the fourth image feature.

[0109] (Supplementary Note 8) In the information processing system of any one of Supplementary Note 1 to Supplementary Note 7, the first loss function is the sum of a logarithm of a conditional confidence of the first image feature conditioned on the third image feature and a logarithm of a conditional reciprocal confidence of the second image feature conditioned on the third image feature, and the second loss function includes a component that uses the logarithm of the conditional confidence of the second image feature conditioned on the third image feature as a generator loss, and a component that uses a first-order norm of the difference between the third image feature and the fourth image feature as the feature loss function.

[0110] (Supplementary Note 9) An information processing method in an information processing system, comprising: a first identification step of identifying a first image feature in a feature region of an original image using a first machine learning model for the original image; a compression step of generating compressed data with a reduced amount of data using a second machine learning model for the original image; a restoration step of generating a restored image of the original image from the compressed data using a third machine learning model; a second identification step of identifying a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image; a third image feature extraction step of extracting a third image feature for object recognition from the original image; and a fourth image feature extraction step of extracting a fourth image feature for object recognition from the restored image. and a model learning step of setting a parameter set of the fourth machine learning model in common with a parameter set of the first machine learning model, determining a parameter set of the first machine learning model so that a first loss function indicating the degree of variation from the conditional reliability of the first image feature conditioned on the third image feature to the conditional reliability of the second image feature conditioned on the third image feature becomes larger, and determining parameter sets of each of the second machine learning model and the third machine learning model so that a second loss function obtained by combining the conditional reliability of the second image feature conditioned on the third image feature and a feature loss function indicating the degree of variation from the third image feature to the fourth image feature becomes smaller.

[0111] (Supplementary Note 10) An information processing device includes a model learning means for determining a parameter set for each of the second and third machine learning models so that a first loss function indicating a degree of variation in the conditional reliability conditioned on the third image feature becomes larger, from a conditional reliability of a first image feature identified in a feature region of the original image using a first machine learning model for the original image, using the original image as a condition, to a conditional reliability of a second image feature identified in a feature region of the restored image, the restored image being generated using a third machine learning model from compressed data with a reduced data volume generated using a second machine learning model for the original image, and determining a parameter set for each of the second and third machine learning models so that a second loss function obtained by combining the conditional reliability of the second image feature conditioned on the third image feature and a feature loss function indicating a degree of variation from the third image feature to a fourth image feature for recognizing the object extracted from the restored image becomes smaller.

[0112] (Appendix 11) A storage medium storing a program for causing a computer to function as the information processing device described in Appendix 10.

[0113] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Additions, omissions, substitutions, and other modifications of the configuration are possible within the scope of the gist of the present invention. The direction of arrows shown in block diagrams and other drawings is for convenience of explanation, and the disclosure of this application does not limit the direction of flow of information, data, signals, etc. during implementation. Furthermore, the present invention is not limited by the foregoing description, but is limited only by the appended claims. [Industrial Applicability]

[0114] According to the information processing system, information processing device, information processing method, and storage medium of each of the above aspects, the restored image obtained using the second machine learning model and the third machine learning has the third image feature as a condition, and has the second image feature that varies significantly from the first image feature, thereby improving visual quality. Furthermore, using the same method as the method for extracting the third image feature from the original image, the fourth image feature can be extracted from the restored image so that the variation from the third image feature is reduced. Therefore, the subjective quality of the restored image and the recognition rate of image recognition using the fourth image feature extracted from the restored image can be improved. [Explanation of symbols]

[0115] 1, 1a, 1c...information processing system, 12...encoding unit, 14...input processing unit, 16...imaging unit, 22...decoding unit, 30...compression processing unit, 32, 32b...first identification unit (first identification means), 34...second identification unit (second identification means), 36...model learning unit (model learning means), 38, 38b...third image feature extraction unit (third image feature extraction means), 39...fourth image feature extraction unit (fourth image feature extraction means), 42...image recognition unit, 44...detection unit, 46...display processing unit, 47...display unit, 48...operation input unit, 124...compression unit (compression means), 224...restoration unit (restoration means), 321...first image feature extraction unit, 322 (322-1 to 322-3)...resampling 324 (324-1 to 324-3)...concatenation unit, 325 (325-1 to 325-3)...convolution processing unit, 326 (326-1 to 326-3)...pooling unit, 327...concatenation unit, 328...normalization unit, 362...data amount calculation unit, 364...feature loss calculation unit, 365...filter setting unit, 366...parameter update unit, 367...filter processing unit, 382-1...first type image feature extraction unit, 382-2...second type image feature extraction unit, 382-3...third type image feature extraction unit, 1242...characteristic analysis unit, 1244...first distribution estimation unit, 1246...first sampling unit, 2242...second distribution estimation unit, 2244...second sampling unit, 2246...data generation unit

Claims

1. a first identification means for identifying a first image feature in a feature region of an original image by using a first machine learning model for the original image; a compression means for generating compressed data with a reduced amount of data from the original image by using a second machine learning model; a restoration means for generating a restored image of the original image from the compressed data using a third machine learning model; a second identification means for identifying a second image feature in a feature region of the restored image by using a fourth machine learning model on the restored image; a third image feature extracting means for extracting a third image feature for recognizing a subject from the original image; a fourth image feature extracting means for extracting a fourth image feature for recognizing the object from the restored image; The parameter set of the fourth machine learning model is common to the parameter set of the first machine learning model; determining a parameter set of the first machine learning model so that a first loss function indicating a degree of variation from a conditional reliability of the first image feature conditioned on the third image feature to a conditional reliability of the second image feature conditioned on the third image feature becomes larger; and a model learning means for determining parameter sets for the second machine learning model and the third machine learning model so that a second loss function obtained by combining a conditional reliability of the second image feature conditional on the third image feature and a feature loss function indicating a degree of variation from the third image feature to the fourth image feature becomes smaller. Information processing system.

2. The second loss function is further combined with an information loss function based on the information amount of the compressed data. The information processing system according to claim 1 .

3. The information loss function is the maximum value between the information amount of the compressed data and the target value of the information amount. The information processing system according to claim 2 .

4. The third image feature and the fourth image feature each include image features for recognizing a plurality of types of subjects. The information processing system according to any one of claims 1 to 3.

5. A filtering means is provided, the filtering means filters the original image with spatial frequency characteristics that differ for each frame to generate a processed image; The model learning means determining a parameter set for the first machine learning model using first image features identified from the processed image, second image features identified from a restored image based on the processed image obtained from compressed data generated from the processed image, and the third image features extracted from the processed image; determining parameter sets for the second machine learning model and the third machine learning model using the first image feature, the second image feature, the third image feature, and a fourth image feature identified from a restored image based on the processed image; The information processing system according to any one of claims 1 to 4.

6. a transmitting device including the first identifying means and the compressing means; a receiving device including the restoration means and the second identification means; parameter notification means, The model learning means a first classification means for learning that classifies the first image feature for the original image; a training compression means for generating the compressed data for the original image; a learning restoration means for generating a restored image of the original image from the compressed data; a second classification means for learning that classifies a second image feature in a feature region of the restored image using a fourth machine learning model for the restored image; a third image feature extracting means for extracting a third image feature for recognizing a subject from the original image; a fourth image feature extracting means for extracting a fourth image feature for recognizing the subject from the restored image, The parameter notification means The parameter set of the first machine learning model, the parameter set of the second machine learning model, the parameter set of the third machine learning model, and the parameter set of the fourth machine learning model determined by the model learning means are notified to the first identification means, the compression means, the restoration means, and the second identification means, respectively. The information processing system according to any one of claims 1 to 5.

7. the first image feature, the second image feature, the third image feature, and the fourth image feature each have a plurality of element values; The first identification means a first image feature extraction means for extracting the first image feature from the original image; a first resampling means for resampling the first image feature so that the number of elements of the first image feature is equal to the number of elements of the third image feature; a first reliability calculation means for calculating a conditional reliability of the first image feature from a first combined image feature obtained by combining the resampled first image feature and the third image feature, The second identification means a second image feature extraction means for extracting the second image feature from the restored image; second resampling means for resampling the second image feature so that the number of elements of the second image feature is equal to the number of elements of the fourth image feature; a second reliability calculation means for calculating a conditional reliability of the second image feature from a second combined image feature obtained by combining the resampled second image feature and the fourth image feature. The information processing system according to any one of claims 1 to 6.

8. An information processing method in an information processing system, comprising: a first identification step of identifying a first image feature in a feature region of an original image using a first machine learning model for the original image; a compression step of generating compressed data having a reduced amount of data from the original image using a second machine learning model; a restoration step of generating a restored image of the original image from the compressed data using a third machine learning model; a second identification step of identifying a second image feature in a feature region of the restored image using a fourth machine learning model on the restored image; a third image feature extraction step of extracting third image features for object recognition from the original image; a fourth image feature extraction step of extracting fourth image features for recognizing the object from the restored image; The parameter set of the fourth machine learning model is common to the parameter set of the first machine learning model; determining a parameter set of the first machine learning model so that a first loss function indicating a degree of variation from a conditional reliability of the first image feature conditioned on the third image feature to a conditional reliability of the second image feature conditioned on the third image feature becomes larger; a model learning step of determining parameter sets for the second machine learning model and the third machine learning model so that a second loss function obtained by combining a conditional reliability of the second image feature conditional on the third image feature and a feature loss function indicating a degree of variation from the third image feature to the fourth image feature becomes smaller. Information processing methods.

9. a parameter set of the first machine learning model is determined so that a first loss function indicating the degree of change from a conditional confidence of a first image feature identified in a feature region of the original image using a first machine learning model for the original image, using the third image feature for recognizing the subject extracted from the original image as a condition, to a conditional confidence of a second image feature identified in a feature region of the restored image using a fourth machine learning model that shares a parameter set with the first machine learning model for a restored image of the original image generated using a third machine learning model from compressed data with a reduced data volume generated using a second machine learning model for the original image, the parameter set of the first machine learning model being greater; a model learning means for determining parameter sets for the second machine learning model and the third machine learning model so as to reduce a second loss function obtained by combining a conditional reliability of the second image feature conditional on the third image feature and a feature loss function indicating the degree of variation from the third image feature to a fourth image feature for recognizing the subject extracted from the restored image; An information processing device comprising:

10. A program for causing a computer to function as the information processing device according to claim 9.

Images