Generation program, inference program, generation method, and information processing device

Abstract

This information processing device acquires captured image data of a subject present in water. The information processing device enters the acquired image data into a machine learning model having a first layer that performs processing for restoring the color of the subject in an air equivalent and a second layer that performs processing for restoring the edge components of the subject, and thereby obtains the output results produced by the machine learning model. The information processing device generates a trained machine learning model on the basis of the acquired output results and correct answer data indicating the subject in the air equivalent.

Description

Generation program, inference program, generation method, and information processing device

[0001] The present invention relates to a generation program and the like.

[0002] To realize ocean digital twins, technology is being used to generate 3D data from underwater images. Such 3D data is used to represent the ocean in digital space.

[0003] In underwater images, the quality is degraded due to the influence of turbidity, such as blue and green color casts and blurred edges, which reduces the information of the subject. For this reason, it is necessary to restore the quality of the subject in the underwater image before generating 3D data.

[0004] For example, conventional techniques for restoring quality related to color cast and edge blur include RUIESR (Realistic Underwater Image Enhancement and Super-resolution), SwinIR, and HAT (Hybrid Attention Transformer).

[0005] RUIESR generates realistic low-resolution images and then segments the resulting images into two types of degradation: resolution degradation and underwater degradation. RUIESR uses the segmentation results as prior knowledge to train and use a model for restoring underwater images. For example, RUIESR uses a model based on a convolutional neural network (CNN).

[0006] SwinIR is an application of the Swin Transformer to super-resolution technology. In SwinIR, feature calculation is performed not only on the windowed segmentation results of the input image using SW-MSA (Shifted Window-based Multi-head Self-Attention), but also on the windowed segmentation results of a Shifted Window, which is a diagonally shifted window.

[0007] In addition to SW-MSA, HAT adds a Hybrid Attention Block (HAB) that calculates color channel (RGB) features using a Channel Attention Block (CAB) in the feature calculation process. HAT also uses Overlapping Cross-Attention (OCA) to divide the input image so that the windows that divide it overlap with other windows, allowing it to utilize more global features.

[0008] Japanese Patent Application Laid-Open No. 2024-013658

[0009] However, the above-mentioned conventional techniques are unable to restore the quality of the subject in the underwater image.

[0010] For example, because only limited information is available for underwater images, it is difficult to perform super-resolution using a learning model based on CNN, which uses local features. Also, SwinIR and HAT are designed to process images in the air, so they are not very effective for underwater images.

[0011] In one aspect, the present invention aims to provide a generation program, an inference program, a generation method, and an information processing device that can restore the quality of a subject in an underwater image.

[0012] In the first proposal, a computer is caused to execute the following process. The computer acquires image data of an object photographed underwater. The computer inputs the acquired image data into a machine learning model having a first layer that performs a process of restoring the color of the object in an air-equivalent state and a second layer that performs a process of restoring the edge components of the object, thereby acquiring an output result output by the machine learning model. The computer generates a trained machine learning model based on the acquired output result and ground truth data that shows the object in an air-equivalent state.

[0013] The quality of the subject in the underwater image can be restored.

[0014] FIG. 1 is a diagram for explaining the processing of an information processing device according to this embodiment. FIG. 2 is a diagram for explaining feature (1) of the processing of the information processing device according to this embodiment. FIG. 3 is a diagram for explaining feature (2) of the processing of the information processing device according to this embodiment. FIG. 4 is a diagram for explaining feature (2) of the processing of the information processing device according to this embodiment. FIG. 5 is a diagram for explaining feature (3) of the processing of the information processing device according to this embodiment. FIG. 6 is a functional block diagram showing the configuration of the information processing device according to this embodiment. FIG. 7 is a flowchart (1) showing the processing procedure of the information processing device according to this embodiment. FIG. 8 is a flowchart (2) showing the processing procedure of the information processing device according to this embodiment. FIG. 9 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device according to the embodiment.

[0015] Hereinafter, embodiments of the generation program, inference program, generation method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0016] The processing of the information processing device according to this embodiment will be described. Fig. 1 is a diagram for explaining the processing of the information processing device according to this embodiment. The information processing device 100 restores an underwater image using an image quality restoration model 50.

[0017] The image quality restoration model 50 is a model that combines a muddyness removal model 50a and a super-resolution model 50b. The muddyness removal model 50a is a machine learning model that restores the original color of the subject. The muddyness removal model 50a corresponds to the "first layer." The muddyness removal model 50a is, for example, a neural network (NN).

[0018] The super-resolution model 50b is a machine learning model that improves blurring of the edges of a subject (restoring edge components). The super-resolution model 50b corresponds to the "second layer." The super-resolution model 50b is, for example, a neural network.

[0019] For example, image data input to the image quality restoration model 50 is input to the turbidity removal model 50a, the output result of the turbidity removal model 50a is input to the super-resolution model 50b, and the output result is output from the super-resolution model 50b. The output result from the super-resolution model 50b becomes the final output result of the image quality restoration model 50.

[0020] The information processing device 100 uses the training data 10 to perform machine learning on the image quality restoration model 50. In the following description, performing machine learning on the image quality restoration model 50 may be referred to as "training the image quality restoration model 50."

[0021] The training data 10 includes input data 10a and correct answer data 10b. The input data 10a is an image of a subject taken underwater (underwater image). The correct answer data 10b is an image of the underwater subject shown in air (air image).

[0022] The information processing device 100 uses the training data 10 to train the image quality restoration model 50 as follows.

[0023] The information processing device 100 generates restored data 20 by inputting input data 10a of the training data 10 into an image quality restoration model 50. The information processing device 100 evaluates the degree of match between the correct answer data 10b of the training data 10 and the restored data 20, and updates the parameters of the image quality restoration model 50 based on the evaluation result of the degree of match.

[0024] The information processing device 100 repeatedly executes the above process using a plurality of pieces of training data included in the training data set.

[0025] Next, the characteristics (1) to (4) of the processing of the information processing apparatus 100 according to this embodiment will be described in order.

[0026] Feature (1) of the processing of the information processing device 100 will be described. Figures 2 and 3 are diagrams for explaining feature (1) of the processing of the information processing device according to this embodiment. When the information processing device 100 executes image quality restoration processing using the image quality restoration model 50, it sets multiple windows in the input image in the super-resolution model 50b and executes feature calculation processing for each window.

[0027] First, a description will be given of Fig. 2. The information processing device 100 performs window division on the input image 30, and sets windows 31-1, 31-2, 31-3, and 31-4 in the input image 30.

[0028] The information processing device 100 sets the windows 32-1, 32-2, 32-3, 32-4, 32-5, and 32-6 by translating the positions of the windows 31-1 to 31-4 in the "vertical direction."

[0029] The information processing device 100 sets the windows 33-1, 33-2, 33-3, 33-4, 33-5, and 33-6 by translating the positions of the windows 31-1 to 31-4 in the "horizontal direction."

[0030] The information processing device 100 sets windows 34-1, 34-2, 34-3, 34-4, 34-5, 34-6, 34-7, 34-8, and 34-9 by translating the positions of the windows 31-1 to 31-4 in the "diagonal direction."

[0031] The information processing apparatus 100 sets the windows 35-1, 35-2, 35-3, and 35-4 by normalizing the number of windows among the windows 32-1 to 32-6.

[0032] The information processing apparatus 100 sets the windows 36-1, 36-2, 36-3, and 36-4 by normalizing the window numbers of the windows 33-1 to 32-6.

[0033] The information processing apparatus 100 sets the windows 37-1, 37-2, 37-3, and 37-4 by normalizing the number of windows among the windows 34-1 to 34-9.

[0034] As described above, the information processing device 100 performs the window division as shown in FIG. 2, whereby 16 types of windows (31-1 to 31-4, 35-1 to 35-4, 36-1 to 36-4, 37-1 to 37-4) are set for the input image 30.

[0035] Now, we move on to the description of Fig. 3. The information processing device 100 performs window size enlargement for each window set in the input image 30 described in Fig. 2. When enlarging a window size, the information processing device 100 enlarges the window size so that the window includes the boundaries of adjacent windows.

[0036] The information processing device 100 enlarges the window size of the window 31-1 and sets a window 38-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 31-2 to 31-4.

[0037] The information processing device 100 enlarges the window size of the window 35-1 and sets a window 39-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 35-2 to 35-4.

[0038] The information processing device 100 enlarges the window size of the window 36-1 and sets the window 40-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 36-2 to 36-4.

[0039] The information processing device 100 enlarges the window size of the window 37-1 and sets a window 41-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 37-2 to 37-4.

[0040] As described above, the information processing device 100 sets 16 more types of windows by enlarging the window size as shown in Fig. 3. That is, the information processing device 100 sets 32 (16 x 2 = 32) types of windows for the input image 30 by performing the processes in Fig. 2 and Fig. 3. The information processing device 100 performs feature calculation processing for each window set for the input image 30 in the super-resolution model 50b.

[0041] By performing the above-described window division and window size expansion, the information processing device 100 can calculate features at all window boundaries, enabling the use of global features compared to conventional techniques. Furthermore, by expanding the window size to include the boundaries, it is possible to reduce the expansion to the minimum necessary and reduce calculation costs.

[0042] For example, the information processing device 100 may use a HAT as the super-resolution model 50b. When using the HAT, the information processing device 100 causes an overlapping cross-attention block (OCAB) included in the HAT to perform the processes (window division, window size enlargement, and feature calculation) described in FIGS. 2 and 3 .

[0043] Next, a description will be given of feature (2) of the processing of the information processing device 100. Fig. 4 is a diagram for explaining feature (2) of the processing of the information processing device according to this embodiment. When performing image quality restoration processing using the image quality restoration model 50, the information processing device 100 performs processing to calculate the importance of each component in multiple color spaces in the super-resolution model 50b.

[0044] For example, the super-resolution model 50b includes a hybrid attention block (HAB) 40 as shown in Fig. 4. The HAB 40 includes LayerNorms 41a and 41b, a multi-layer perceptron (MLP) 42, a CAB (RGB) 43, an RGB2HSV 44, a CAB (HSV) 45, an (S)W-MSA 46, and adders 47a and 47b.

[0045] The LayerNorm 41a normalizes the input information and outputs the normalized information to the CAB(RGB) 43, RGB2HSV 44, and (S)W-MSA 46.

[0046] The CAB (RGB) 43 is a cross-attention block that calculates the importance of each of R, G, and B. The CAB (RGB) 43 outputs the calculation results to the adder 47.

[0047] The RGB2HSV 44 converts the input RGB values ​​into HSV (Hue Saturation Value) values, and outputs the converted information to the CAB (HSV) 45.

[0048] The CAB (HSV) 45 is a cross-attention block that calculates the importance of each of H, S, and V. The CAB (HSV) 45 outputs the calculation results to the adder 47.

[0049] A shifted window-based multi-head self-attention (S)W-MSA 46 calculates a weight for each position and outputs the calculation result to an adder 47.

[0050] The adder 47a performs element-wise summation based on the input information, and outputs the calculation result to the LayerNorm 41b and the adder 47b.

[0051] The LayerNorm 41b performs normalization on the input information and outputs the normalized information to the MLP 42.

[0052] The MLP 42 is a type of basic neural network in which multiple layers are combined in series. The MLP 42 processes the input information and outputs the processing result to the adder 47b.

[0053] The adder 47b performs element-wise summation based on the input information, and outputs the calculation result to the next block.

[0054] As described above, the information processing apparatus 100 calculates the importance in a plurality of color spaces as described with reference to FIG. 4, thereby making it possible to effectively utilize the global characteristics of colors.

[0055] Next, feature (3) of the processing of the information processing device 100 will be described. Fig. 5 is a diagram for explaining feature (3) of the processing of the information processing device according to this embodiment. As described in Fig. 1, the information processing device 100 evaluates the degree of match between the supervised answer data 10b of the training data 10 and the restored data 20, and updates the parameters of the image quality restoration model 50 based on the evaluation result of the degree of match. Here, the information processing device 100 further uses a classifier 55 when evaluating the degree of match.

[0056] The classifier 55 receives input data 10a, correct answer data 10b, and restored data 20. The classifier 55 outputs a value between 0 and 1. If the input is deemed genuine, a value close to 1 is output, and if the input is deemed fake, a value close to 0 is output. The classifier 55 is an NN or the like.

[0057] The information processing device 100 performs evaluation using adversarial loss by using the output result of the image quality restoration model 50 and the output result of the classifier 55 that identifies ground truth data in air or equivalent to air.

[0058] More specifically, the information processing device 100 calculates the loss function L shown in Equation (1) GAN The degree of match is evaluated by L included in Eq. G is a loss function for the image quality restoration model 50, and is expressed by equation (2). D is the loss function for the classifier 55 and is given by equation (3).

[0059]

[0060] G(x) shown in equation (2) is a value output from the image quality restoration model 50 when input data 10a is input to the image quality restoration model 50. For example, G(x) corresponds to restored data 20. D(G(x)) shown in equation (2) is a value output from the classifier 55 when G(x) is input to the classifier 55.

[0061] D(y) shown in equation (3) is a value output from the classifier 55 when the correct answer data 10b is input to the classifier 55. D(x) shown in equation (3) is a value output from the classifier 55 when the input data 10a is input to the classifier 55. The explanation regarding D(G(x)) is the same as the explanation above.

[0062] The information processing device 100 calculates the loss function L shown in Equation (1). GANThe parameters of the image quality restoration model 50 and the classifier 55 are updated (trained) so as to reduce the value of . For example, the information processing device trains the image quality restoration model 50 and the classifier 55 based on the backpropagation method.

[0063] As described above, the information processing device 100 not only treats the ground truth data as real and the restoration results of the image quality restoration model 50 as fakes, but also uses underwater images (input data) as fakes and incorporates them into the adversarial loss function. This makes it possible to impose a large penalty on images that include underwater features, thereby improving restoration accuracy.

[0064] Next, feature (4) of the processing of the information processing device 100 will be described. As described in FIG. 1, the information processing device 100 evaluates the degree of match between the supervised data 10b of the training data 10 and the restored data 20, and updates the parameters of the image quality restoration model 50 based on the evaluation result of the degree of match. In this evaluation, the information processing device 100 evaluates the degree of match between the image quality restoration processing result and the supervised data in multiple color spaces. For example, the multiple color spaces are RGB and HSV.

[0065] For example, the information processing device 100 calculates the loss function L shown in Equation (4) color The degree of match is evaluated by L included in Equation (4). RGB is a loss function for the color space RGB, and is expressed by equation (5). HSV is the loss function for the color space HSV and is defined by equation (6).

[0066]

[0067] y included in equation (5) R is the value of R of the correct answer data 10b. R is the value of R in the restored data 20. G is the value of G of the correct answer data 10b. G is the value of G in the restored data 20. B is the value of B in the correct answer data 10b. B is the value of B in the restored data 20.

[0068] y included in equation (6) H is the value of H of the correct answer data 10b. H is the value of H in the restored data 20. S is the value of S of the correct answer data 10b. S is the value of S in the restored data 20. V is the value of V of the correct answer data 10b. V is the value of V in the restored data 20.

[0069] As described above, the information processing device 100 uses RGB and HSV as information on multiple color spaces used in the loss function. In particular, HSV is a color space that is close to human vision, and by calculating the feature amounts of a color space that is close to human vision, it is possible to train the image quality restoration model 50 by making better use of global color features.

[0070] The above has described the processing features (1) to (4) of the information processing device 100 according to this embodiment. Note that the information processing device 100 may use either one of the processing features (3) or (4) to train the image quality restoration model 50, or may use both of them to train the image quality restoration model 50.

[0071] Next, a configuration example of an information processing device 100 that executes the above-described processing will be described. Fig. 6 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 6, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0072] The communication unit 110 executes data communication with external devices via a network, and is realized by a network interface card (NIC) or the like.

[0073] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0074] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0075] The storage unit 140 includes the image quality restoration model 50, the classifier 55, and a training data set 141. The storage unit 140 is a memory or the like.

[0076] The image quality restoration model 50 is a model that combines a turbidity removal model 50a and a super-resolution model 50b. When an underwater image is input to the image quality restoration model 50, restored data is output in which the subject in the underwater image is restored to an equivalent subject in the air. Other explanations regarding the image quality restoration model 50 are the same as those regarding the image quality restoration model 50 in FIG. 1.

[0077] The classifier 55 is a model that outputs a value between 0 and 1 when it receives training data (input data, correct answer data) and restored data output from the image quality restoration model 50. Other explanations regarding the classifier 55 are the same as those regarding the classifier 55 in FIG.

[0078] The teacher data set 141 includes a plurality of teacher data. The explanation of each teacher data is the same as the explanation of the teacher data 10 in FIG.

[0079] The control unit 150 includes an acquisition unit 151, a learning unit 152, and an inference unit 153. The control unit 150 is, for example, a CPU (Central Processing Unit) or the like.

[0080] The acquisition unit 151 acquires various types of information from the communication unit 110 or the input unit 120. For example, the acquisition unit 151 may acquire the teacher dataset 141 or image data to be inferred from an external device or the like.

[0081] The learning unit 152 reads the image quality restoration model 50 and the classifier 55 and trains the image quality restoration model 50 .

[0082] For example, the learning unit 152 acquires training data from the training data set 141, and based on the training data, repeatedly executes a process of training the image quality restoration model 50. The training process executed by the learning unit 152 is similar to the process described with reference to FIGS.

[0083] The inference unit 153 performs restoration of underwater images based on the image quality restoration model 50 trained by the learning unit 152. For example, the inference unit 153 acquires image data of the inference target, inputs the acquired image data into the trained image quality restoration model 50, and obtains an output result of the image quality restoration model 50. The inference unit 153 outputs the output result as restored image data. The inference unit 153 may output the restored image data to an external device or the like that executes processing related to the marine digital twin.

[0084] Next, an example of the processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 7 is a flowchart (1) showing the processing procedure of the information processing device according to this embodiment. Fig. 7 shows the procedure in which the information processing device 100 trains the image quality restoration model 50.

[0085] As shown in FIG. 7, the learning unit 152 of the information processing device 100 acquires training data (input data and correct answer data) from the training data set 141 (step S101).

[0086] The learning unit 152 generates restored data by inputting the input data to the image quality restoration model 50 (step S102). The learning unit 152 inputs the input data, the correct answer data, and the restored data to the classifier 55, thereby obtaining an output result from the classifier 55 (step S103).

[0087] The learning unit 152 calculates the loss function L GAN and the loss function L color The value of is calculated and integrated into the value of the loss function L according to equation (7) (step S104). α and β in equation (7) are predefined weighting coefficients. The learning unit 152 updates the parameters of the image quality restoration model 50 and the classifier 55 so as to reduce the value of the loss function L (step S105).

[0088]

[0089] If the learning unit 152 does not want to end the training (step S106, No), the learning unit 152 proceeds to step S101. On the other hand, if the learning unit 152 wants to end the training (step S106, Yes), the learning unit 152 outputs the trained image quality restoration model 50 (step S107).

[0090] 8 is a flowchart (2) showing the processing procedure of the information processing device according to this embodiment. In FIG. 8, the information processing device 100 restores image data using the trained image quality restoration model 50.

[0091] The inference unit 153 of the information processing device 100 reads the trained image quality restoration model 50 (step S201).

[0092] The inference unit 153 acquires image data to be restored (step S202), and inputs the image data to be restored to the trained image quality restoration model 50 (step S203).

[0093] The inference unit 153 outputs the output result of the trained image quality restoration model 50 as restored image data (step S204).

[0094] The inference unit 153 repeatedly executes the above process every time image data to be restored is acquired.

[0095] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 generates a trained image quality restoration model 50 based on input data of an underwater subject and ground truth data showing the subject in an air-equivalent state. By inputting image data to be restored into the image quality restoration model 50, the quality of the subject in the underwater image can be restored.

[0096] The image quality restoration model 50 trained by the information processing device 100 performs window division on the input image, and further sets multiple windows by moving the multiple windows in multiple directions and enlarging each window, and performs feature calculation for each window set in the input image. By performing window division and window size enlargement in this way, feature calculation becomes possible at all window boundaries, and global features can be utilized compared to conventional techniques. Furthermore, by enlarging the window size to include the boundaries, it is possible to reduce calculation costs while keeping the enlargement to the minimum necessary.

[0097] The image quality restoration model 50 trained by the information processing device 100 calculates importance in multiple color spaces (RGB and HSV), thereby making effective use of global color features.

[0098] The information processing device 100 further uses the classifier 55 to train the image quality restoration model 50. This not only treats the ground truth data as real and the restoration results of the image quality restoration model 50 as fake, but also uses underwater images (input data) as fakes and incorporates them into the adversarial loss function, making it possible to impose a large penalty on images that include underwater features and improve restoration accuracy.

[0099] The information processing device 100 uses RGB and HSV as information on multiple color spaces used in the loss function. In particular, HSV is a color space that is close to human vision, and by calculating the feature amounts of a color space that is close to human vision, it is possible to train the image quality restoration model 50 by making better use of global color features.

[0100] The information processing device 100 restores image data by inputting image data to be restored to the trained image quality restoration model 50. In this way, the quality of the subject in the underwater image can be restored.

[0101] Furthermore, the information processing device 100 can construct a digital twin using the restored image data, which is the output result of the inference unit 153. In other words, the restored image data is data for constructing a digital twin that virtually reproduces the state of the ocean in real space. For example, the information processing device 100 places 3D data generated from the restored image data on the digital twin that virtually reproduces the state of the ocean in real space. For example, the information processing device 100 places 3D data of pipes installed underwater or coral reefs on the digital twin. In this way, the information processing device 100 can create a digital twin that reproduces the 3D shape data of objects that exist in the ocean.

[0102] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 shown in the above embodiment will be described in order.

[0103] 9 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. As shown in FIG. 9, the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also includes a communication device 204 that exchanges data with external devices via a wired or wireless network, and an interface device 205. A microphone, a speaker, and the like may be connected to the interface device 205. The computer 200 also includes a RAM 206 that temporarily stores various information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0104] The hard disk drive 207 stores an acquisition program 207 a, a learning program 207 b, and an inference program 207 c. The CPU 201 reads out each of the programs 207 a to 207 c and loads them into the RAM 206.

[0105] The acquisition program 207a functions as the acquisition process 206a, the learning program 207b functions as the learning process 206b, and the inference program 207c functions as the inference process 206c.

[0106] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the learning process 206b corresponds to the processing of the learning unit 152. The processing of the inference process 206c corresponds to the processing of the inference unit 153.

[0107] It should be noted that each of the programs 207a to 207c does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207c.

[0108] 50 Image quality restoration model 55 Classifier 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 150 Control unit 151 Acquisition unit 152 Learning unit 153 Inference unit

Claims

1. A generation program that causes a computer to execute the following process: acquire image data of a subject in water; input the acquired image data into a machine learning model having a first layer that performs a process of restoring the color of the subject in an air-equivalent state and a second layer that performs a process of restoring the edge components of the subject; obtain an output result output by the machine learning model; and generate a trained machine learning model based on the acquired output result and ground truth data that shows the subject in an air-equivalent state.

2. The generation program of claim 1, wherein the second layer divides the image data into a plurality of first windows, sets a plurality of second windows by translating the plurality of first windows in a plurality of directions, sets a plurality of third windows by enlarging the plurality of first windows, sets a plurality of fourth windows by enlarging the plurality of second windows, and calculates features of the image data based on the plurality of first windows, the plurality of second windows, the plurality of third windows, and the plurality of fourth windows.

3. The generation program according to claim 1, characterized in that the second layer calculates the importance of RGB (Red Green Blue) and the importance of HSV (Hue Saturation Value) based on the image data.

4. The generation program of claim 1, further comprising a classifier that receives the image data, the correct answer data, and the output result output by the machine learning model as input, and outputs a value corresponding to the degree of match between the image data and the correct answer data and the degree of match between the correct answer data and the output result, wherein the process of generating the trained machine learning model further utilizes the value output from the classifier to generate the trained machine learning model.

5. The generation program described in claim 1, characterized in that the process of generating the trained machine learning model generates the trained machine learning model based on the RGB values ​​and HSV values ​​contained in the output result output by the machine learning model and the RGB values ​​and HSV values ​​contained in the correct answer data.

6. An inference program that causes a computer to execute the following process: input image data of a subject in water to be restored, the image data being an image of a subject in water, into the machine learning model, which is trained using image data of a subject in water as an explanatory variable and ground truth data showing the subject in air as a target variable, the machine learning model having a first layer that executes a process of restoring the color of the subject in air and a second layer that executes a process of restoring the edge components of the subject; and output the output result from the machine learning model as restored image data.

7. A generation method characterized by a computer executing the following process: acquiring image data of a subject photographed in water; inputting the acquired image data into a machine learning model having a first layer that performs a process of restoring the color of the subject in an air-equivalent state and a second layer that performs a process of restoring the edge components of the subject; obtaining an output result output by the machine learning model; and generating a trained machine learning model based on the acquired output result and ground truth data that shows the subject in an air-equivalent state.

8. The generation method described in claim 7, characterized in that the second layer divides the image data into a plurality of first windows, sets a plurality of second windows by translating the plurality of first windows in a plurality of directions, sets a plurality of third windows by enlarging the plurality of first windows, sets a plurality of fourth windows by enlarging the plurality of second windows, and calculates features of the image data based on the plurality of first windows, the plurality of second windows, the plurality of third windows, and the plurality of fourth windows.

9. The generation method according to claim 7, characterized in that the second layer calculates the importance of RGB (Red Green Blue) and the importance of HSV (Hue Saturation Value) based on the image data.

10. The method of generating the trained machine learning model described in claim 7, further comprising a classifier that receives the image data, the correct answer data, and the output result output by the machine learning model as inputs, and outputs a value corresponding to the degree of match between the image data and the correct answer data and the degree of match between the correct answer data and the output result, wherein the process of generating the trained machine learning model further utilizes the value output from the classifier to generate the trained machine learning model.

11. The generation method described in claim 7, characterized in that the process of generating the trained machine learning model generates the trained machine learning model based on the RGB values ​​and HSV values ​​contained in the output result output by the machine learning model and the RGB values ​​and HSV values ​​contained in the correct answer data.

12. An information processing device having a control unit that executes the following process: acquires image data of a subject photographed in water; inputs the acquired image data into a machine learning model having a first layer that executes a process of restoring the color of the subject in an air-equivalent state and a second layer that executes a process of restoring the edge components of the subject; obtains an output result output by the machine learning model; and generates a trained machine learning model based on the acquired output result and ground truth data that shows the subject in an air-equivalent state.

13. The information processing device described in claim 12, characterized in that the second layer divides the image data into a plurality of first windows, sets a plurality of second windows by translating the plurality of first windows in a plurality of directions, sets a plurality of third windows by enlarging the plurality of first windows, sets a plurality of fourth windows by enlarging the plurality of second windows, and calculates features of the image data based on the plurality of first windows, the plurality of second windows, the plurality of third windows, and the plurality of fourth windows.

14. The information processing device according to claim 12, wherein the second layer calculates the importance of RGB (Red Green Blue) and the importance of HSV (Hue Saturation Value) based on the image data.

15. The information processing device of claim 12, further comprising a classifier that receives the image data, the correct answer data, and the output result output by the machine learning model as input, and outputs a value corresponding to the degree of match between the image data and the correct answer data and the degree of match between the correct answer data and the output result, wherein the process of generating the trained machine learning model further utilizes the value output from the classifier to generate the trained machine learning model.

16. The information processing device described in claim 12, characterized in that the process of generating the trained machine learning model generates the trained machine learning model based on the RGB values ​​and HSV values ​​contained in the output result output by the machine learning model and the RGB values ​​and HSV values ​​contained in the correct answer data.

Images