Generation program, inference program, generation method, and information processing device

Abstract

This information processing device acquires image data obtained by imaging a subject present in the water. The information processing device inputs the acquired image data to a machine learning model which has a first layer that executes processing for restoring the colors of a subject in an air equivalent, and a second layer that executes processing for restoring edge components of the subject, and thereby acquires an output result output by the machine learning model. The information processing device generates a trained machine learning model on the basis of the acquired output result and correct answer data showing the subject in the air equivalent.

Description

Generation program, inference program, generation method, and information processing device

[0001] The present invention relates to a generation program and the like.

[0002] To realize ocean digital twins, technology is being used to generate 3D data from underwater images. Such 3D data is used to represent the ocean in digital space.

[0003] In underwater images, the quality is degraded due to the influence of turbidity, such as blue and green color casts and blurred edges, which reduces the information of the subject. For this reason, it is necessary to restore the quality of the subject in the underwater image before generating 3D data.

[0004] For example, conventional techniques for restoring quality related to color cast and edge blur include RUIESR (Realistic Underwater Image Enhancement and Super-Resolution), SwinIR, and HAT (Hybrid Attention Transformer).

[0005] RUIESR generates realistic low-resolution images and then segments the resulting images into two types of degradation: resolution degradation and underwater degradation. RUIESR uses the segmentation results as prior knowledge to train and use a model for restoring underwater images. For example, RUIESR uses a model based on a convolutional neural network (CNN).

[0006] SwinIR is an application of the Swin Transformer to super-resolution technology. In SwinIR, feature calculation is performed not only on the windowed segmentation results of the input image using SW-MSA (Shifted Window-based Multi-head Self-Attention), but also on the windowed segmentation results of a Shifted Window, which is a diagonally shifted window.

[0007] In addition to SW-MSA, HAT adds a Hybrid Attention Block (HAB) that calculates color channel (RGB) features using a Channel Attention Block (CAB) in the feature calculation process. HAT also uses Overlapping Cross-Attention (OCA) to divide the input image so that the windows that divide it overlap with other windows, allowing it to utilize more global features.

[0008] Japanese Patent Application Laid-Open No. 2024-013658

[0009] However, the above-mentioned conventional techniques are unable to restore the quality of the subject in the underwater image.

[0010] For example, because only limited information is available for underwater images, it is difficult to perform super-resolution using a learning model based on CNN, which uses local features. Also, SwinIR and HAT are designed to process images in the air, so they are not very effective for underwater images.

[0011] In one aspect, the present invention aims to provide a generation program, an inference program, a generation method, and an information processing device that can restore the quality of a subject in an underwater image.

[0012] In the first proposal, a computer is caused to execute the following process. The computer acquires image data of an object photographed underwater. The computer inputs the acquired image data into a machine learning model having a first layer that performs a process of restoring the color of the object in an air-equivalent state and a second layer that performs a process of restoring the edge components of the object, thereby acquiring an output result output by the machine learning model. The computer generates a trained machine learning model based on the acquired output result and ground truth data that shows the object in an air-equivalent state.

[0013] The quality of the subject in the underwater image can be restored.

[0014] FIG. 1 is a diagram for explaining processing of an information processing device according to the first embodiment. FIG. 2 is a diagram for explaining feature (1) of processing of an information processing device according to the first embodiment. FIG. 3 is a diagram for explaining feature (2) of processing of an information processing device according to the first embodiment. FIG. 4 is a diagram for explaining feature (2) of processing of an information processing device according to the first embodiment. FIG. 5 is a diagram for explaining feature (3) of processing of an information processing device according to the first embodiment. FIG. 6 is a functional block diagram showing a configuration of an information processing device according to the first embodiment. FIG. 7 is a flowchart for explaining a processing procedure of an information processing device according to the first embodiment. FIG. 8 is a flowchart for explaining a processing procedure of an information processing device according to the first embodiment. FIG. 9 is a diagram showing an example of conventional SEM-SEG processing. FIG. 10 is a diagram for explaining an example of additional learning in the conventional technology. FIG. 11 is a diagram for explaining processing of testing a SEM-SEG additional learning model in the conventional technology. FIG. 12 is a diagram for explaining processing of an information processing device according to the second embodiment. FIG. 13 is a diagram for explaining processing of an information processing device according to the second embodiment. FIG. 14 is a diagram for explaining a second evaluation. FIG. 15 is a diagram for explaining the effect of the information processing device according to the second embodiment. FIG. 16 is a functional block diagram showing the configuration of the information processing device according to the second embodiment. FIG. 17 is a flowchart showing the processing procedure of additional learning. FIG. 18 is a flowchart showing the processing procedure of two types of evaluation. FIG. 19 is a flowchart showing the processing procedure of SEMASEG processing. FIG. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiments.

[0015] Hereinafter, embodiments of the generation program, inference program, generation method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0016] The following describes the processing of the information processing device according to this embodiment 1. Fig. 1 is a diagram for explaining the processing of the information processing device according to this embodiment 1. The information processing device 100 uses an image quality restoration model 50 to restore an underwater image.

[0017] The image quality restoration model 50 is a model that combines a muddyness removal model 50a and a super-resolution model 50b. The muddyness removal model 50a is a machine learning model that restores the original color of the subject. The muddyness removal model 50a corresponds to the "first layer." The muddyness removal model 50a is, for example, a neural network (NN).

[0018] The super-resolution model 50b is a machine learning model that improves blurring of the edges of a subject (restoring edge components). The super-resolution model 50b corresponds to the "second layer." The super-resolution model 50b is, for example, a neural network.

[0019] For example, image data input to the image quality restoration model 50 is input to the turbidity removal model 50a, the output result of the turbidity removal model 50a is input to the super-resolution model 50b, and the output result is output from the super-resolution model 50b. The output result from the super-resolution model 50b becomes the final output result of the image quality restoration model 50.

[0020] The information processing device 100 uses the training data 10 to perform machine learning on the image quality restoration model 50. In the following description, performing machine learning on the image quality restoration model 50 may be referred to as "training the image quality restoration model 50."

[0021] The training data 10 includes input data 10a and correct answer data 10b. The input data 10a is an image of a subject taken underwater (underwater image). The correct answer data 10b is an image of the underwater subject shown in air (air image).

[0022] The information processing device 100 uses the training data 10 to train the image quality restoration model 50 as follows.

[0023] The information processing device 100 generates restored data 20 by inputting input data 10a of the training data 10 into an image quality restoration model 50. The information processing device 100 evaluates the degree of match between the correct answer data 10b of the training data 10 and the restored data 20, and updates the parameters of the image quality restoration model 50 based on the evaluation result of the degree of match.

[0024] The information processing device 100 repeatedly executes the above process using a plurality of pieces of training data included in the training data set.

[0025] Next, the characteristics (1) to (4) of the processing of the information processing apparatus 100 according to the first embodiment will be described in order.

[0026] The following describes feature (1) of the processing of the information processing device 100. Figures 2 and 3 are diagrams for explaining feature (1) of the processing of the information processing device according to the first embodiment. When the information processing device 100 executes image quality restoration processing using the image quality restoration model 50, it sets multiple windows in the input image in the super-resolution model 50b and executes feature calculation processing for each window.

[0027] First, a description will be given of Fig. 2. The information processing device 100 performs window division on the input image 30, and sets windows 31-1, 31-2, 31-3, and 31-4 in the input image 30.

[0028] The information processing device 100 sets the windows 32-1, 32-2, 32-3, 32-4, 32-5, and 32-6 by translating the positions of the windows 31-1 to 31-4 in the "vertical direction."

[0029] The information processing device 100 sets the windows 33-1, 33-2, 33-3, 33-4, 33-5, and 33-6 by translating the positions of the windows 31-1 to 31-4 in the "horizontal direction."

[0030] The information processing device 100 sets windows 34-1, 34-2, 34-3, 34-4, 34-5, 34-6, 34-7, 34-8, and 34-9 by translating the positions of the windows 31-1 to 31-4 in the "diagonal direction."

[0031] The information processing apparatus 100 sets the windows 35-1, 35-2, 35-3, and 35-4 by normalizing the number of windows among the windows 32-1 to 32-6.

[0032] The information processing apparatus 100 sets the windows 36-1, 36-2, 36-3, and 36-4 by normalizing the window numbers of the windows 33-1 to 32-6.

[0033] The information processing apparatus 100 sets the windows 37-1, 37-2, 37-3, and 37-4 by normalizing the number of windows among the windows 34-1 to 34-9.

[0034] As described above, the information processing device 100 performs the window division as shown in FIG. 2, whereby 16 types of windows (31-1 to 31-4, 35-1 to 35-4, 36-1 to 36-4, 37-1 to 37-4) are set for the input image 30.

[0035] Now, we move on to the description of Fig. 3. The information processing device 100 performs window size enlargement for each window set in the input image 30 described in Fig. 2. When enlarging a window size, the information processing device 100 enlarges the window size so that the window includes the boundaries of adjacent windows.

[0036] The information processing device 100 enlarges the window size of the window 31-1 and sets a window 38-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 31-2 to 31-4.

[0037] The information processing device 100 enlarges the window size of the window 35-1 and sets a window 39-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 35-2 to 35-4.

[0038] The information processing device 100 enlarges the window size of the window 36-1 and sets the window 40-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 36-2 to 36-4.

[0039] The information processing device 100 enlarges the window size of the window 37-1 and sets a window 41-1. Similarly, the information processing device 100 enlarges the window sizes of the windows 37-2 to 37-4.

[0040] As described above, the information processing device 100 sets 16 more types of windows by enlarging the window size as shown in Fig. 3. That is, the information processing device 100 sets 32 (16 x 2 = 32) types of windows for the input image 30 by performing the processes in Fig. 2 and Fig. 3. The information processing device 100 performs feature calculation processing for each window set for the input image 30 in the super-resolution model 50b.

[0041] By performing the above-described window division and window size expansion, the information processing device 100 can calculate features at all window boundaries, enabling the use of global features compared to conventional techniques. Furthermore, by expanding the window size to include the boundaries, it is possible to reduce the expansion to the minimum necessary and reduce calculation costs.

[0042] For example, the information processing device 100 may use a HAT as the super-resolution model 50b. When using the HAT, the information processing device 100 causes an overlapping cross-attention block (OCAB) included in the HAT to perform the processes (window division, window size enlargement, and feature calculation) described in FIGS. 2 and 3 .

[0043] Next, a description will be given of feature (2) of the processing of the information processing device 100. Fig. 4 is a diagram for explaining feature (2) of the processing of the information processing device according to the first embodiment. When the information processing device 100 executes image quality restoration processing using the image quality restoration model 50, the information processing device 100 executes processing to calculate the importance of each component in a plurality of color spaces in the super-resolution model 50b.

[0044] For example, the super-resolution model 50b includes a hybrid attention block (HAB) 40 as shown in Fig. 4. The HAB 40 includes LayerNorms 41a and 41b, a multi-layer perceptron (MLP) 42, a CAB (RGB) 43, an RGB2HSV 44, a CAB (HSV) 45, an (S)W-MSA 46, and adders 47a and 47b.

[0045] The LayerNorm 41a normalizes the input information and outputs the normalized information to the CAB(RGB) 43, RGB2HSV 44, and (S)W-MSA 46.

[0046] The CAB (RGB) 43 is a cross-attention block that calculates the importance of each of R, G, and B. The CAB (RGB) 43 outputs the calculation results to the adder 47.

[0047] The RGB2HSV 44 converts the input RGB values ​​into HSV (Hue Saturation Value) values, and outputs the converted information to the CAB (HSV) 45.

[0048] The CAB (HSV) 45 is a cross-attention block that calculates the importance of each of H, S, and V. The CAB (HSV) 45 outputs the calculation results to the adder 47.

[0049] A shifted window-based multi-head self-attention (S)W-MSA 46 calculates a weight for each position and outputs the calculation result to an adder 47.

[0050] The adder 47a performs element-wise summation based on the input information, and outputs the calculation result to the LayerNorm 41b and the adder 47b.

[0051] The LayerNorm 41b performs normalization on the input information and outputs the normalized information to the MLP 42.

[0052] The MLP 42 is a type of basic neural network in which multiple layers are combined in series. The MLP 42 processes the input information and outputs the processing result to the adder 47b.

[0053] The adder 47b performs element-wise summation based on the input information, and outputs the calculation result to the next block.

[0054] As described above, the information processing apparatus 100 calculates the importance in a plurality of color spaces as described with reference to FIG. 4, thereby making it possible to effectively utilize the global characteristics of colors.

[0055] Next, feature (3) of the processing of the information processing device 100 will be described. Fig. 5 is a diagram for explaining feature (3) of the processing of the information processing device according to the first embodiment. As described in Fig. 1, the information processing device 100 evaluates the degree of match between the supervised answer data 10b of the training data 10 and the restored data 20, and updates the parameters of the image quality restoration model 50 based on the evaluation result of the degree of match. Here, the information processing device 100 further uses a classifier 55 when evaluating the degree of match.

[0056] The classifier 55 receives input data 10a, correct answer data 10b, and restored data 20. The classifier 55 outputs a value between 0 and 1. If the input is deemed genuine, a value close to 1 is output, and if the input is deemed fake, a value close to 0 is output. The classifier 55 is an NN or the like.

[0057] The information processing device 100 performs evaluation using adversarial loss by using the output result of the image quality restoration model 50 and the output result of the classifier 55 that identifies ground truth data in air or equivalent to air.

[0058] More specifically, the information processing device 100 calculates the loss function L shown in Equation (1) GAN The degree of match is evaluated by L included in Eq. G is a loss function for the image quality restoration model 50, and is expressed by equation (2). D is the loss function for the classifier 55 and is given by equation (3).

[0059]

[0060] G(x) shown in equation (2) is a value output from the image quality restoration model 50 when input data 10a is input to the image quality restoration model 50. For example, G(x) corresponds to restored data 20. D(G(x)) shown in equation (2) is a value output from the classifier 55 when G(x) is input to the classifier 55.

[0061] D(y) shown in equation (3) is a value output from the classifier 55 when the correct answer data 10b is input to the classifier 55. D(x) shown in equation (3) is a value output from the classifier 55 when the input data 10a is input to the classifier 55. The explanation regarding D(G(x)) is the same as the explanation above.

[0062] The information processing device 100 calculates the loss function L shown in Equation (1). GAN The parameters of the image quality restoration model 50 and the classifier 55 are updated (trained) so as to reduce the value of . For example, the information processing device trains the image quality restoration model 50 and the classifier 55 based on the backpropagation method.

[0063] As described above, the information processing device 100 not only treats the ground truth data as real and the restoration results of the image quality restoration model 50 as fakes, but also uses underwater images (input data) as fakes and incorporates them into the adversarial loss function. This makes it possible to impose a large penalty on images that include underwater features, thereby improving restoration accuracy.

[0064] Next, feature (4) of the processing of the information processing device 100 will be described. As described in FIG. 1, the information processing device 100 evaluates the degree of match between the supervised data 10b of the training data 10 and the restored data 20, and updates the parameters of the image quality restoration model 50 based on the evaluation result of the degree of match. In this evaluation, the information processing device 100 evaluates the degree of match between the image quality restoration processing result and the supervised data in multiple color spaces. For example, the multiple color spaces are RGB and HSV.

[0065] For example, the information processing device 100 calculates the loss function L shown in Equation (4) color The degree of match is evaluated by L included in Equation (4). RGB is a loss function for the color space RGB, and is expressed by equation (5). HSV is the loss function for the color space HSV and is defined by equation (6).

[0066]

[0067] y included in equation (5) R is the value of R of the correct answer data 10b. R is the value of R in the restored data 20. G is the value of G of the correct answer data 10b. G is the value of G in the restored data 20. B is the value of B in the correct answer data 10b. B is the value of B in the restored data 20.

[0068] y included in equation (6) H is the value of H of the correct answer data 10b. H is the value of H in the restored data 20. S is the value of S of the correct answer data 10b. S is the value of S in the restored data 20. V is the value of V of the correct answer data 10b. V is the value of V in the restored data 20.

[0069] As described above, the information processing device 100 uses RGB and HSV as information on multiple color spaces used in the loss function. In particular, HSV is a color space that is close to human vision, and by calculating the feature amounts of a color space that is close to human vision, it is possible to train the image quality restoration model 50 by making better use of global color features.

[0070] The above has described the processing features (1) to (4) of the information processing device 100 according to the present embodiment 1. Note that the information processing device 100 may use either one of the processing features (3) or (4) to train the image quality restoration model 50, or may use both of them to train the image quality restoration model 50.

[0071] Next, a configuration example of the information processing device 100 that executes the above-described processing will be described. Fig. 6 is a functional block diagram showing the configuration of the information processing device according to the first embodiment. As shown in Fig. 6, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0072] The communication unit 110 executes data communication with external devices via a network, and is realized by a network interface card (NIC) or the like.

[0073] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0074] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0075] The storage unit 140 includes the image quality restoration model 50, the classifier 55, and a training data set 141. The storage unit 140 is a memory or the like.

[0076] The image quality restoration model 50 is a model that combines a turbidity removal model 50a and a super-resolution model 50b. When an underwater image is input to the image quality restoration model 50, restored data is output in which the subject in the underwater image is restored to an equivalent subject in the air. Other explanations regarding the image quality restoration model 50 are the same as those regarding the image quality restoration model 50 in FIG. 1.

[0077] The classifier 55 is a model that outputs a value between 0 and 1 when it receives training data (input data, correct answer data) and restored data output from the image quality restoration model 50. Other explanations regarding the classifier 55 are the same as those regarding the classifier 55 in FIG.

[0078] The teacher data set 141 includes a plurality of teacher data. The explanation of each teacher data is the same as the explanation of the teacher data 10 in FIG.

[0079] The control unit 150 includes an acquisition unit 151, a learning unit 152, and an inference unit 153. The control unit 150 is, for example, a CPU (Central Processing Unit) or the like.

[0080] The acquisition unit 151 acquires various types of information from the communication unit 110 or the input unit 120. For example, the acquisition unit 151 may acquire the teacher dataset 141 or image data to be inferred from an external device or the like.

[0081] The learning unit 152 reads the image quality restoration model 50 and the classifier 55 and trains the image quality restoration model 50 .

[0082] For example, the learning unit 152 acquires training data from the training data set 141, and based on the training data, repeatedly executes a process of training the image quality restoration model 50. The training process executed by the learning unit 152 is similar to the process described with reference to FIGS.

[0083] The inference unit 153 performs restoration of underwater images based on the image quality restoration model 50 trained by the learning unit 152. For example, the inference unit 153 acquires image data of the inference target, inputs the acquired image data into the trained image quality restoration model 50, and obtains an output result of the image quality restoration model 50. The inference unit 153 outputs the output result as restored image data. The inference unit 153 may output the restored image data to an external device or the like that executes processing related to the marine digital twin.

[0084] Next, an example of a processing procedure of the information processing device 100 according to the present embodiment 1 will be described. Fig. 7 is a flowchart (1) showing the processing procedure of the information processing device according to the present embodiment 1. Fig. 7 shows the procedure in which the information processing device 100 trains the image quality restoration model 50.

[0085] As shown in FIG. 7, the learning unit 152 of the information processing device 100 acquires training data (input data and correct answer data) from the training data set 141 (step S101).

[0086] The learning unit 152 generates restored data by inputting the input data to the image quality restoration model 50 (step S102). The learning unit 152 inputs the input data, the correct answer data, and the restored data to the classifier 55, thereby obtaining an output result from the classifier 55 (step S103).

[0087] The learning unit 152 calculates the loss function LGAN and the loss function L color The value of is calculated and integrated into the value of the loss function L according to equation (7) (step S104). α and β in equation (7) are predefined weighting coefficients. The learning unit 152 updates the parameters of the image quality restoration model 50 and the classifier 55 so as to reduce the value of the loss function L (step S105).

[0088]

[0089] If the learning unit 152 does not want to end the training (step S106, No), the learning unit 152 proceeds to step S101. On the other hand, if the learning unit 152 wants to end the training (step S106, Yes), the learning unit 152 outputs the trained image quality restoration model 50 (step S107).

[0090] 8 is a flowchart (2) illustrating the processing procedure of the information processing device according to the present embodiment 1. In FIG. 8, the information processing device 100 illustrates the procedure for restoring image data using the trained image quality restoration model 50.

[0091] The inference unit 153 of the information processing device 100 reads the trained image quality restoration model 50 (step S201).

[0092] The inference unit 153 acquires image data to be restored (step S202), and inputs the image data to be restored to the trained image quality restoration model 50 (step S203).

[0093] The inference unit 153 outputs the output result of the trained image quality restoration model 50 as restored image data (step S204).

[0094] The inference unit 153 repeatedly executes the above process every time image data to be restored is acquired.

[0095] Next, the effects of the information processing device 100 according to the first embodiment will be described. The information processing device 100 generates a trained image quality restoration model 50 based on input data of an image of a subject existing underwater and ground truth data showing the subject in an air-equivalent state. By inputting image data to be restored into the image quality restoration model 50, the quality of the subject in the underwater image can be restored.

[0096] The image quality restoration model 50 trained by the information processing device 100 performs window division on the input image, and further sets multiple windows by moving the multiple windows in multiple directions and enlarging each window, and performs feature calculation for each window set in the input image. By performing window division and window size enlargement in this way, feature calculation becomes possible at all window boundaries, and global features can be utilized compared to conventional techniques. Furthermore, by enlarging the window size to include the boundaries, it is possible to reduce calculation costs while keeping the enlargement to the minimum necessary.

[0097] The image quality restoration model 50 trained by the information processing device 100 calculates importance in multiple color spaces (RGB and HSV), thereby making effective use of global color features.

[0098] The information processing device 100 further uses the classifier 55 to train the image quality restoration model 50. This not only treats the ground truth data as real and the restoration results of the image quality restoration model 50 as fake, but also uses underwater images (input data) as fakes and incorporates them into the adversarial loss function, making it possible to impose a large penalty on images that include underwater features and improve restoration accuracy.

[0099] The information processing device 100 uses RGB and HSV as information on multiple color spaces used in the loss function. In particular, HSV is a color space that is close to human vision, and by calculating the feature amounts of a color space that is close to human vision, it is possible to train the image quality restoration model 50 by making better use of global color features.

[0100] The information processing device 100 restores image data by inputting image data to be restored to the trained image quality restoration model 50. In this way, the quality of the subject in the underwater image can be restored.

[0101] Furthermore, the information processing device 100 can construct a digital twin using the restored image data, which is the output result of the inference unit 153. In other words, the restored image data is data for constructing a digital twin that virtually reproduces the state of the ocean in real space. For example, the information processing device 100 places 3D data generated from the restored image data on the digital twin that virtually reproduces the state of the ocean in real space. For example, the information processing device 100 places 3D data of pipes installed underwater or coral reefs on the digital twin. In this way, the information processing device 100 can create a digital twin that reproduces the 3D shape data of objects that exist in the ocean.

[0102] An underwater image is, for example, an image taken underwater. In the ocean, the presence of organic matter absorbs light with short wavelengths. Therefore, as the concentration of organic matter increases in the ocean, the amount of blue light becomes relatively small, causing the ocean to appear green or brown. In addition, the ocean appears muddy when soil and mud from the land flow into it from rivers or when mud from the seabed is stirred up. The subject is, for example, an object that exists underwater. For example, the object is an underwater structure such as an offshore power plant or a pipe. For example, the object is a living organism such as a coral reef or seaweed.

[0103] The information processing device 100 inputs image data to be restored, which is an image of an object existing in the sea, and outputs the output result from the machine learning model as restored image data. At this time, the restored image data converts a first color of the subject, which has become different from the color expected in air due to the influence of organic matter or mud present in the sea, into a second color of the subject equivalent to the color in air. In addition, the restored image data eliminates blurring of the subject's edges due to the influence of organic matter, mud, or ocean currents present in the sea.

[0104] Furthermore, the information processing device 100 places the 3D data generated from the restored image data in the digital twin. More specifically, the information processing device 100 places the subject object in the digital twin, in which the subject is defined by a second color and the blurring of the subject's edges has been eliminated. In other words, the subject object placed in the digital twin is in a state in which the subject's color is defined by the second color and the subject's edges are not blurred. This allows the information processing device 100 to place the subject object that exists in the sea in the digital twin in a distinguishable state.

[0105] Here, an example of a process for generating an ocean digital twin by the information processing device 100 will be described. For example, the information processing device 100 reproduces ocean conditions, such as temperature and ocean currents, and the creatures that exist in the ocean under those conditions in the digital twin. First, the information processing device 100 generates a digital twin that reproduces the real-world underwater conditions in a virtual space using the detection results of a sensor that senses the ocean conditions. Next, the information processing device 100 acquires image data captured underwater and determines whether the acquired image data contains an object. If it is determined that an object is included, the information processing device 100 inputs image data to be restored, which captures the object as the subject, and outputs the output result from the machine learning model as restored image data. Next, the information processing device 100 uses the restored image data to generate an object of the subject, in which the subject is defined with a second color and the blurring of the subject's edges is eliminated. Then, the information processing device 100 places the generated object in the virtual space of the digital twin.

[0106] Furthermore, the information processing device 100 associates the state of the ocean on the digital twin where the subject object is placed with the object and outputs the associated information to the display screen. More specifically, the information processing device 100 associates information indicating the ocean temperature and ocean currents at the position on the digital twin where the object is placed with an image of the object and displays the associated information on the screen of the display device.

[0107] This allows the information processing device 100 to arrange the objects of interest present in the ocean in a distinguishable manner on the digital twin in accordance with the ocean conditions. Furthermore, the information processing device 100 can, for example, allow the user to visually confirm what types of creatures exist in the ocean based on the underwater conditions.

[0108] Before describing the information processing device according to the second embodiment, an example of a conventional technique for semantic segmentation will be described. In the following description, semantic segmentation will be referred to as "SemaSeg" where appropriate.

[0109] 9 is a diagram showing an example of conventional SemaSeg processing. As shown in FIG. 9, when SemaSeg is performed on image Im10, SemaSeg result Im11 is obtained. In SemaSeg result Im11, a label is associated with every pixel in the image. For example, a pixel corresponding to a car is associated with the label "car." Other pixels are similarly associated with corresponding labels.

[0110] For example, when performing SemaSeg processing on an image such as that described in FIG. 9 , a SemaSeg model trained using a large-scale dataset is used. However, when performing SemaSeg processing on an underwater image such as that described in Example 1 (an image in which quality degradation occurs due to blue or green color casts and edge blurring caused by turbidity), it is difficult to directly use a SemaSeg model trained using a large-scale dataset. In addition, the quality of underwater images is low and there is little variation. For this reason, in conventional technology, additional learning is performed on a SemaSeg model trained using a large-scale dataset, as described in FIG. 10 .

[0111] Fig. 10 is a diagram illustrating an example of additional learning using conventional technology. In the description of Fig. 10, a device that performs processing related to the conventional technology will be referred to as a "conventional device." The conventional device prepares a SEM-SEG additional learning model 60b by copying a SEM-SEG model 60a that has been trained using a large-scale dataset, and performs additional learning on the SEM-SEG additional learning model 60b using teacher data 61.

[0112] The SemaSeg model 60a and the SemaSeg additional learning model 60b are NNs, etc. For example, when an image is input to the trained SemaSeg model 60a, a SemaSeg processing result is obtained in which a label is assigned to each pixel of the image.

[0113] The training data 61 includes input data 61 a and correct answer data 61 b. The input data 61 a is an image of a subject taken underwater (underwater image). The correct answer data 61 b ​​is correct answer data obtained by performing SEM-Seg processing on the input data 61 a.

[0114] The conventional device uses the teacher data 61 to perform additional learning of the SEM-SEG additional learning model 60b as follows.

[0115] The conventional device inputs input data 61a of teacher data 61 into a SemaSeg additional learning model 60b to generate a SemaSeg processing result 62. The conventional device evaluates the degree of match between correct answer data 61b of teacher data 61 and the SemaSeg processing result 62, and updates the parameters of the SemaSeg additional learning model 60b based on the evaluation result of the degree of match.

[0116] The conventional device repeatedly executes the above process using a plurality of pieces of training data included in the training data set. After the additional training of the SEM-SEG additional training model 60b is completed, the conventional device tests the SEM-SEG additional training model 60b as shown in FIG.

[0117] 11 is a diagram illustrating the process of testing a SEM SEG additional learning model in the prior art. As shown in FIG. 11, the prior art device inputs test data (underwater images) 63 into a trained SEM SEG additional learning model 60b to obtain a SEM SEG processing result 64.

[0118] As described above, in the conventional technology, a SemaSegment model that has been trained on a large-scale dataset is subjected to additional learning using underwater images, and the additionally trained SemaSegment additional learning model 60b is used to perform SemaSegment on underwater images.

[0119] However, with conventional technology, the quality and subject variation in the training data used for additional learning is insufficient, making it impossible to perform SemaSeg processing with high accuracy. For example, comparing the test data 63 in Figure 11 with the SemaSeg processing result 64, it is clear that the fish region is not properly labeled.

[0120] Next, a process of an information processing device according to the second embodiment will be described. The information processing device according to the second embodiment will be referred to as an "information processing device 200." FIGS. 12 and 13 are diagrams for explaining the process of the information processing device according to the second embodiment.

[0121] First, a description will be given of Fig. 12. The information processing device 200 has an additional learning model 56 that is a copy of the image quality restoration model 50 trained by the information processing device 100 of Example 1. The additional learning model 56 is a neural network or the like. The information processing device 200 also has a SMART model 60a that has been trained using a large-scale data set. The information processing device 200 performs additional learning on the additional learning model 56 using training data 70.

[0122] The training data 70 includes input data 70a and correct answer data 70b. The input data 70a is an image of a subject taken underwater (underwater image). The correct answer data 70b is correct answer data obtained when SEM-Segment processing is performed on the input data 70a.

[0123] The information processing device 200 generates restored data 70c by inputting input data 70a of the teacher data 70 to the additional learning model 56. The information processing device 200 inputs the restored data 70c to the SEM-SEG model 60a to obtain a SEM-SEG processing result 70d.

[0124] The information processing device 200 performs two types of evaluation based on the correct answer data 70b and the SEM-SEG processing result, and updates the parameters of the additional learning model 56 so as to improve the evaluation result.

[0125] The information processing device 200 repeatedly executes the above process using a plurality of pieces of training data included in the training data set.

[0126] Next, we move on to the description of Fig. 13. In Fig. 13, the information processing device 200 performs SEM processing on test data (underwater images) using the additionally trained model 56 and the SEM model 60a.

[0127] The information processing device 200 generates restored data 81 by inputting test data (underwater images) 80 into the incremental learning model 56. The information processing device 200 obtains a SEM-SEG processing result 82 by inputting the restored data 81 into the SEM-SEG model 60a.

[0128] Here, the "two types of evaluation" performed by the information processing device 200 described in Fig. 12 will be described in more detail. For example, the two types of evaluation include a first evaluation related to improving the performance of SEM and a second evaluation for monitoring damage to a subject captured in an image.

[0129] The first evaluation is performed by calculating the degree of agreement between the SemaSeg processing result (for example, the SemaSeg processing result 70d in FIG. 12) and the correct data (for example, the correct data in FIG. 12) as an evaluation value using cross-entropy.

[0130] For example, the information processing device 200 calculates the evaluation value L by the first evaluation based on the formula (8). performance Calculate "y ｉ " is the result of the SemaSeg processing. ｉ (Hat)" is the correct data. Evaluation value L performance is "y ｉ " and "y ｉ The closer the match is, the smaller the value becomes. Evaluation value L performance A smaller means improved SemaSeg performance.

[0131]

[0132] Next, the second evaluation will be described. Fig. 14 is a diagram for explaining the second evaluation. The information processing device 200 uses a CLIP (Contrastive Language-Image Pre-Training) model 57. For example, when image data is input to the CLIP model 57, a vector of the image data is output. When a label (character string) of the correct data is input to the CLIP model 57, a vector of the label is output.

[0133] For example, the information processing device 200 inputs the restored data 70c into the CLIP model 57 to obtain the vector v img The information processing device 200 inputs the label “starfish” assigned to the correct answer data 70b into the CLIP model 57 to calculate the vector v text If the restored data 70c contains an image of a "starfish", the vector v img and the vector v text A high similarity means that no corruption has occurred in the restored data 70c generated in the additional learning model 56.

[0134] The information processing device 200 calculates the vector v based on the equation (9). img and the vector v text Furthermore, the information processing device 200 calculates the similarity (cosine similarity) between the first and second evaluation values ​​L based on the formula (10). monitor Calculate the vector v img and the vector v text The higher the similarity with monitor becomes smaller.

[0135]

[0136] The first evaluation and the second evaluation have been described above. The information processing device 200 calculates an evaluation value L that combines the first evaluation and the second evaluation. For example, the evaluation value L is defined as shown in Equation (11). α and β in Equation (11) are weights (fixed values) that are set in advance.

[0137]

[0138] The information processing device 200 updates the parameters of the additional learning model 56 so as to reduce the value of the evaluation value L. Improvement of the two types of evaluation results described above means that the value of the evaluation value L decreases.

[0139] Next, an example of a process in which the information processing device 200 updates the parameters of the additional learning model 56 based on two types of evaluation results (evaluation value L) will be described.

[0140] The information processing device 200 calculates the gradient ∇L of the evaluation value L based on equation (12). In equation (12), w represents a parameter of the additional learning model 56. N represents the number of parameter elements. If the gradient ∇L is positive, it indicates that the evaluation result has deteriorated. On the other hand, if the gradient ∇L is negative, it indicates that the evaluation result has improved. Note that the information processing device 200 uses the backpropagation method when calculating equation (12).

[0141]

[0142] The information processing device 200 uses equation (13) when updating the parameters of the additional learning model 56. In equation (13), w t are the current parameters of the incremental learning model 56. t+1 are the updated parameters of the incremental learning model 56. ε is the learning rate, which is a fixed value defined in advance.

[0143]

[0144] The information processing device 200 calculates the gradient ∇L of the evaluation value L based on equation (12), and repeatedly executes the process of updating the parameter w of the additional learning model 56 based on equation (13).

[0145] The above describes an example of processing by the information processing device 200. As described above, before performing SEM-Seg processing, the information processing device 200 adds the additional learning model 56 (image quality restoration model 50) and performs additional learning based on two types of evaluation on the additional learning model 56. This makes it possible to provide an additional learning technique that resolves the lack of quality of training data and the lack of variation in subjects.

[0146] The information processing device 200 also generates SMART processing results by inputting underwater images into the additionally trained model 56 to generate restored data, and inputting the restored data into the SMART model 60a. This enables SMART processing to be performed on underwater images with high accuracy.

[0147] 15 is a diagram for explaining the effect of the information processing device according to the second embodiment. When SMART processing is performed on test data 63 using conventional technology, a SMART processing result 64 is obtained. On the other hand, the information processing device 200 inputs the test data 63 into the additionally trained additional learning model 56 to generate restored data, and inputs the restored data into the SMART model 60a to obtain a SMART processing result 65. Comparing the SMART processing result 64 and the SMART processing result 65, it can be seen that the SMART processing result 65 by the information processing device 200 is more accurate than the SMART processing result 64 of the conventional technology.

[0148] Next, a configuration example of an information processing device 200 that executes the above-described processing will be described. Fig. 16 is a functional block diagram showing the configuration of an information processing device according to Example 2. As shown in Fig. 16, the information processing device 200 includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0149] The communication unit 210 executes data communication with external devices via a network, and is realized by a NIC or the like.

[0150] The input unit 220 is an input device that inputs various types of information to the information processing device 200. The input unit 220 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0151] The display unit 230 is a display device that displays information output from the control unit 250. The display unit 230 corresponds to a liquid crystal display, an organic EL display, a touch panel, or the like.

[0152] The storage unit 240 includes the image quality restoration model 50, the incremental learning model 56, the CLIP model 57, the SEM-SEG model 60a, and a teacher data set 241. The storage unit 240 is a memory or the like.

[0153] The image quality restoration model 50 is the image quality restoration model 50 described in Example 1. The image quality restoration model 50 is a model trained by the learning unit 152.

[0154] The additional learning model 56 is a model obtained by copying the image quality restoration model 50 .

[0155] The CLIP model 57 is the CLIP model 57 described with reference to FIG.

[0156] The SEM-SEG model 60a is a SEM-SEG model that has been trained in advance using a large-scale data set. Note that the large-scale data set used to train the SEM-SEG model 60a does not include training data of underwater images.

[0157] The teacher data set 241 includes multiple (e.g., N) pieces of teacher data. The teacher data included in the teacher data set 241 is, for example, the teacher data 70 described in FIG. 12. That is, the teacher data includes input data (underwater images) and correct answer data. The correct answer data is the correct answer data obtained when SEM-Segment processing is performed on the input data, and the label of the subject included in the input data is set in the correct answer data. For example, if the input data is input data 70a shown in FIG. 12, the label set in the correct answer data is "starfish."

[0158] The control unit 250 includes an acquisition unit 251, an additional learning unit 252, and a SEM-SEG processing unit 253. The control unit 250 is, for example, a CPU or the like.

[0159] The acquisition unit 251 acquires various types of information from the communication unit 210 or the input unit 220. For example, the acquisition unit 251 may acquire the teacher dataset 241 or image data (underwater images) to be subjected to SEM-SEG processing from an external device, etc. The acquisition unit 251 may acquire the image quality restoration model 50 from the information processing device 100.

[0160] 12, the additional learning unit 252 performs additional learning on the additional learning model 56. The additional learning unit 252 performs the first evaluation and second evaluation described above, and updates the parameters of the additional learning model 56 so that the evaluation value L becomes smaller.

[0161] The SemaSeg processing unit 253 performs SemaSeg processing on image data (underwater images) based on the SemaSeg model 60a and the additional learning model 56 that has been additionally trained by the additional learning unit 252. The processing related to the SemaSeg processing unit 253 corresponds to the processing described in Fig. 13. While Fig. 13 shows an example of performing SemaSeg processing on test data, similar processing may be performed on image data (underwater images) instead of test data.

[0162] The SMART SEG processing unit 253 may output the SMART SEG processing result to the display unit 230 for display, or may notify an external device of the SMART SEG processing result via the communication unit 210.

[0163] Next, an example of a processing procedure of the information processing device 200 according to the second embodiment will be described. First, the processing procedure of the additional learning executed by the information processing device 200 will be described. FIG. 17 is a flowchart showing the processing procedure of the additional learning. As shown in FIG. 17, the acquisition unit 251 of the information processing device 200 acquires the image quality restoration model 50, the SEM-SEG model 60a, and the teacher dataset 241 (step S301). The acquisition unit 251 acquires two types of evaluation weights α and β (step S302).

[0164] The additional learning unit 252 of the information processing device 200 generates the additional learning model 56 by copying the image quality restoration model 50 (step S303). The additional learning unit 252 sets n=1 (step S304).

[0165] The additional learning unit 252 generates restored data by inputting the input data of the n-th teacher data to the additional learning model 56 (step S305). The additional learning unit 252 generates a SEM-SEG processing result by inputting the restored data to the SEM-SEG model 60a (step S306).

[0166] The additional learning unit 252 performs two types of evaluation (step S307). The additional learning unit 252 updates the parameters of the additional learning model 56 so that the evaluation value L based on the two types of evaluation becomes smaller (step S308).

[0167] If the condition n=N is not satisfied (step S309, No), the additional learning unit 252 updates n to n=n+1 (step S311), and proceeds to step S305.

[0168] On the other hand, if the condition n=N is satisfied (Yes at step S309), the additional learning unit 252 outputs the trained additional learning model 56 (step S310).

[0169] Next, the processing procedure for the two types of evaluation shown in step S307 of Fig. 17 will be described in more detail. Fig. 18 is a flowchart showing the processing procedure for the two types of evaluation. As shown in Fig. 18, the additional learning unit 252 of the information processing device 200 acquires restored data, SEM SEG processing results, correct data, labels of the correct data, and weights α and β of the two types of evaluation (step S401).

[0170] The additional learning unit 252 calculates the evaluation value L by the first evaluation based on the formula (8). performance (Step S402). The additional learning unit 252 calculates the vector v of the restored data using the CLIP model 57. img (Step S403). The additional learning unit 252 calculates the label vector v of the correct answer data using the CLIP model 57. text is calculated (step S404).

[0171] The additional learning unit 252 calculates the vector v img , vector v text Based on the similarity calculation result and equation (9), the additional learning unit 252 calculates the evaluation value L monitor is calculated (step S406).

[0172] The additional learning unit 252 calculates the evaluation value L based on equation (11) (step S407), and outputs the evaluation value L (step S408).

[0173] Next, a description will be given of the processing procedure of the SemaSegment processing executed by the information processing device 200. Fig. 19 is a flowchart showing the processing procedure of the SemaSegment processing. As shown in Fig. 19, the SemaSegment processing unit 253 of the information processing device 200 acquires the additionally trained additional learning model 56, the SemaSegment model 60a, and the data to be processed (underwater image) (step S501).

[0174] The SemaSeg processing unit 253 generates restored data by inputting the processing target data to the additional learning model 56 (step S502). The SemaSeg processing unit 253 generates a SemaSeg processing result by inputting the restored data to the SemaSeg model 60a (step S503). The SemaSeg processing unit 253 outputs the SemaSeg processing result (step S504).

[0175] Next, the effects of the information processing device 200 according to the second embodiment will be described. Before performing SEM-SEG processing, the information processing device 200 adds an additional learning model 56 and performs additional learning based on two types of evaluation on the additional learning model 56. This makes it possible to provide an additional learning technique that resolves issues with the quality of training data and the lack of subject variation.

[0176] The information processing device 200 performs two types of evaluations: a first evaluation and a second evaluation. By performing the first evaluation, additional learning can be performed so that the degree of agreement between the SEM-SEG processing result and the correct data is improved by cross-entropy. By performing the second evaluation, additional learning can be performed so that the restored data 70c generated in the additional learning model 56 is not corrupted.

[0177] The information processing device 200 can improve the accuracy of the SemaSegment results by performing SemaSegment processing of underwater images using the additionally learned model 56 and the SemaSegment model 60a.

[0178] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 (200) shown in the above embodiment will be described in order.

[0179] FIG. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. As shown in FIG. 20, the computer 300 has a CPU 301 that executes various arithmetic processes, an input device 302 that accepts data input from a user, and a display 303. The computer 300 also has a communication device 304 that exchanges data with external devices via a wired or wireless network, and an interface device 305. A microphone, a speaker, and the like may be connected to the interface device 305. The computer 300 also has a RAM 306 that temporarily stores various information, and a hard disk drive 307. The devices 301 to 307 are connected to a bus 308.

[0180] The hard disk drive 307 stores an acquisition program 307a, a learning program 307b, an inference program 307c, and an additional learning program 307d. The CPU 301 stores an additional learning program 307d and a SEM-SEG processing program 307e. The CPU 301 reads out each of the programs 307a to 307e and loads them into the RAM 306.

[0181] The acquisition program 307a functions as the acquisition process 306a. The learning program 307b functions as the learning process 306b. The inference program 307c functions as the inference process 306c. The additional learning program 307d functions as the additional learning process 306d. The SemaSeg processing program 307e functions as the SemaSeg processing process 306e.

[0182] The processing of the acquisition process 306a corresponds to the processing of the acquisition units 151, 252. The processing of the learning process 306b corresponds to the processing of the learning unit 152. The processing of the inference process 306c corresponds to the processing of the inference unit 153. The processing of the additional learning process 306d corresponds to the processing of the additional learning unit 252. The processing of the SEM-SEG processing process 306e corresponds to the processing of the SEM-SEG processing unit 253.

[0183] It should be noted that each of the programs 307a to 307e does not necessarily have to be stored in the hard disk drive 307 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307e.

[0184] 50 Image quality restoration model 55 Classifier 56 Additional learning model 57 CLIP model 60a SemaSeg model 100, 200 Information processing device 110, 210 Communication unit 120, 220 Input unit 130, 230 Display unit 140, 240 Storage unit 141, 241 Teacher data set 150, 250 Control unit 151, 251 Acquisition unit 152 Learning unit 153 Inference unit 252 Additional learning unit 253 SemaSeg processing unit

Claims

1. A generation program that causes a computer to execute the following process: acquire image data of a subject in water; input the acquired image data into a machine learning model having a first layer that performs a process of restoring the color of the subject in an air-equivalent state and a second layer that performs a process of restoring the edge components of the subject; obtain an output result output by the machine learning model; and generate a trained machine learning model based on the acquired output result and ground truth data that shows the subject in an air-equivalent state.

2. The generation program of claim 1, wherein the second layer divides the image data into a plurality of first windows, sets a plurality of second windows by translating the plurality of first windows in a plurality of directions, sets a plurality of third windows by enlarging the plurality of first windows, sets a plurality of fourth windows by enlarging the plurality of second windows, and calculates features of the image data based on the plurality of first windows, the plurality of second windows, the plurality of third windows, and the plurality of fourth windows.

3. The generation program according to claim 1, characterized in that the second layer calculates the importance of RGB (Red Green Blue) and the importance of HSV (Hue Saturation Value) based on the image data.

4. The generation program of claim 1, further comprising a classifier that receives the image data, the correct answer data, and the output result output by the machine learning model as input, and outputs a value corresponding to the degree of match between the image data and the correct answer data and the degree of match between the correct answer data and the output result, wherein the process of generating the trained machine learning model further utilizes the value output from the classifier to generate the trained machine learning model.

5. The generation program described in claim 1, characterized in that the process of generating the trained machine learning model generates the trained machine learning model based on the RGB values ​​and HSV values ​​contained in the output result output by the machine learning model and the RGB values ​​and HSV values ​​contained in the correct answer data.

6. The generation program of claim 1, further causing a computer to perform the following processes: inputting image data of an underwater subject into the generated machine learning model to generate restored image data; inputting the restored image data into a trained semantic segmentation model that performs semantic segmentation on image data to generate semantic segmentation results for the restored image data; performing evaluation based on the restored image data, the semantic segmentation results, and ground truth data for semantic segmentation of the image data of the underwater subject; and performing additional learning on the machine learning model based on the evaluation results.

7. The generation program described in claim 6, characterized in that the evaluation process evaluates the degree of agreement between the results of the semantic segmentation and the correct data of the semantic segmentation of image data photographing the subject existing underwater.

8. The generation program described in claim 7, characterized in that the evaluation process further evaluates the similarity between a vector based on the restored image data and a vector of a label assigned to the correct data.

9. An inference program that causes a computer to execute the following process: input image data of a subject in water to be restored, the image data being an image of a subject in water, into the machine learning model, which is trained using image data of a subject in water as an explanatory variable and ground truth data showing the subject in air as a target variable, the machine learning model having a first layer that executes a process of restoring the color of the subject in air and a second layer that executes a process of restoring the edge components of the subject; and output the output result from the machine learning model as restored image data.

10. The inference program described in claim 9, further causing the computer to execute a process of generating semantic segmentation results for the restored image data by inputting the restored image data into a trained semantic segmentation model that performs semantic segmentation on the image data.

11. The inference program described in claim 10, characterized in that the machine learning model is a model that has undergone additional learning based on evaluation based on the restored image data, the results of the semantic segmentation, and correct answer data of the semantic segmentation of image data of an underwater subject, and the input process inputs image data to be restored, which is an image of an underwater subject, to the machine learning model that has undergone the additional learning.

12. The inference program described in claim 9, characterized in that the underwater area is the sea, and the restored image data has a first color of the subject that has become different from the color imagined in air due to the influence of organic matter or mud present in the sea converted into a second color of the subject equivalent to the color in air, and blurring of the edges of the subject present in the sea eliminated.

13. The inference program described in claim 12, characterized in that the restored image data is data for constructing a digital twin that virtually reproduces the state of the ocean in real space, and the subject object placed in the digital twin has a color defined by a second color and does not have blurred edges of the subject.

14. The inference program described in claim 13, characterized in that the subject object is placed on a digital twin that reproduces the state of the ocean in the real space in a virtual space, and the state of the ocean on the digital twin at the position where the object is placed is correlated with an image of the object and displayed on a display screen.

15. A generation method characterized by a computer executing the following process: acquiring image data of a subject in water; inputting the acquired image data into a machine learning model having a first layer that performs a process of restoring the color of the subject in an air-equivalent state and a second layer that performs a process of restoring the edge components of the subject; obtaining an output result output by the machine learning model; and generating a trained machine learning model based on the acquired output result and ground truth data that shows the subject in an air-equivalent state.

16. The generation method described in claim 15, characterized in that the second layer divides the image data into a plurality of first windows, sets a plurality of second windows by translating the plurality of first windows in a plurality of directions, sets a plurality of third windows by enlarging the plurality of first windows, and sets a plurality of fourth windows by enlarging the plurality of second windows, and calculates features of the image data based on the plurality of first windows, the plurality of second windows, the plurality of third windows, and the plurality of fourth windows.

17. The generation method according to claim 15, wherein the second layer calculates the importance of RGB (Red Green Blue) and the importance of HSV (Hue Saturation Value) based on the image data.

18. The method of generating the trained machine learning model described in claim 15, further comprising a classifier that receives the image data, the correct answer data, and the output result output by the machine learning model as inputs, and outputs a value corresponding to the degree of match between the image data and the correct answer data and the degree of match between the correct answer data and the output result, wherein the process of generating the trained machine learning model further utilizes the value output from the classifier to generate the trained machine learning model.

19. The generation method described in claim 15, characterized in that the process of generating the trained machine learning model generates the trained machine learning model based on the RGB values ​​and HSV values ​​contained in the output result output by the machine learning model and the RGB values ​​and HSV values ​​contained in the correct answer data.

20. The method of generating the machine learning model according to claim 15, further comprising the steps of: generating restored image data by inputting image data of an underwater subject into the generated machine learning model; inputting the restored image data into a trained semantic segmentation model that performs semantic segmentation on image data to generate semantic segmentation results for the restored image data; evaluating the restored image data, the semantic segmentation results, and ground truth data for semantic segmentation of the image data of the underwater subject; and performing additional learning on the machine learning model based on the evaluation results.

21. The generation method described in claim 20, characterized in that the evaluation process evaluates the degree of agreement between the results of the semantic segmentation and the correct data of the semantic segmentation of image data of the subject existing in the water.

22. The generation method described in claim 21, characterized in that the evaluation process further evaluates the similarity between a vector based on the restored image data and a vector of a label assigned to the correct data.

23. An information processing device having a control unit that executes the following process: acquires image data of a subject photographed in water; inputs the acquired image data into a machine learning model having a first layer that executes a process of restoring the color of the subject in an air-equivalent state and a second layer that executes a process of restoring the edge components of the subject; obtains an output result output by the machine learning model; and generates a trained machine learning model based on the acquired output result and ground truth data that shows the subject in an air-equivalent state.

24. The information processing device described in claim 23, characterized in that the second layer divides the image data into a plurality of first windows, sets a plurality of second windows by translating the plurality of first windows in a plurality of directions, sets a plurality of third windows by enlarging the plurality of first windows, sets a plurality of fourth windows by enlarging the plurality of second windows, and calculates features of the image data based on the plurality of first windows, the plurality of second windows, the plurality of third windows, and the plurality of fourth windows.

25. The information processing device according to claim 23, wherein the second layer calculates the importance of RGB (Red Green Blue) and the importance of HSV (Hue Saturation Value) based on the image data.

26. The information processing device of claim 23, further comprising a classifier that receives the image data, the correct answer data, and the output result output by the machine learning model as input, and outputs a value corresponding to the degree of match between the image data and the correct answer data and the degree of match between the correct answer data and the output result, wherein the process of generating the trained machine learning model further utilizes the value output from the classifier to generate the trained machine learning model.

27. The information processing device described in claim 23, characterized in that the process of generating the trained machine learning model generates the trained machine learning model based on the RGB values ​​and HSV values ​​contained in the output result output by the machine learning model and the RGB values ​​and HSV values ​​contained in the correct answer data.

28. The information processing device of claim 23, wherein the control unit further performs the following processes: generating restored image data by inputting image data of an underwater subject into the generated machine learning model; generating semantic segmentation results for the restored image data by inputting the restored image data into a trained semantic segmentation model that performs semantic segmentation on image data; performing evaluation based on the restored image data, the semantic segmentation results, and ground truth data for semantic segmentation of the image data of the underwater subject; and performing additional learning on the machine learning model based on the evaluation results.

29. The information processing device described in claim 28, characterized in that the evaluation process evaluates the degree of agreement between the results of the semantic segmentation and the correct data of the semantic segmentation of image data of the subject existing in the water.

30. An information processing device according to claim 29, characterized in that the evaluation process further evaluates the similarity between a vector based on the restored image data and a vector of a label assigned to the correct data.

Images