Model training device, control method, and program

By intensively extracting positive example patches from specific regions and using patchwise loss calculations, the model training device enhances image transformation models' accuracy in converting images between different environments, addressing the limitations of existing methods.

JP7593502B2Active Publication Date: 2024-12-03NEC CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023546668
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-09-10
Publication Date
2024-12-03
Estimated Expiration
2041-09-10

AI Technical Summary

Technical Problem

Existing image transformation models do not adequately consider the influence of specific regions within input images during training, leading to suboptimal performance in converting images between different environments.

Method used

A model training device and method that intensively extracts positive example patches from specific regions of input images, using patchwise loss calculations to enhance the accuracy of feature extraction and image generation between different environmental conditions.

Benefits of technology

The approach allows for high-accuracy conversion of images between environments, such as daytime to nighttime scenes, by focusing on specific objects, thereby improving model performance and enabling robust detection models across varying conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593502000007
    Figure 0007593502000007
  • Figure 0007593502000008
    Figure 0007593502000008
  • Figure 0007593502000009
    Figure 0007593502000009
Patent Text Reader

Abstract

This model training device trains an image conversion model so that an output image representing a scene in a second environment is generated from an input image representing a scene in a first environment. The image conversion model has a feature extraction model for generating a feature map from the input image, and an image generation model for generating an output image from the feature map. The model training device inputs a training image to the image conversion model to thereby obtain a first feature map and an output image, and furthermore inputs the output image to the image conversion model to thereby obtain a second feature map. The model training device calculates a patchwise loss using feature quantities respectively corresponding to a positive example patch extracted from the training image and positive example patches extracted from the positive example patch and the output image, and trains the image conversion model on the basis of the patchwise loss. The positive example patches are extracted in a focused manner from a region representing a specific type of object. 
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to techniques for training models for performing transformations of images. [Background technology]

[0002] A model that generates another image based on an input image, that is, a model that converts an image, has been developed. For example, Patent Document 1 discloses a model that converts an input image into an image of another class, such as converting an image of a horse into an image of a zebra. This model is composed of an encoder that extracts a feature map from an image, and a generator that generates an image from the feature map generated by the encoder. The model is trained based on a loss that is calculated so that the higher the similarity between the feature amount extracted from the positive example region of the input image and the feature amount extracted from the positive example region of the output image, the smaller the loss, and the higher the similarity between the feature amount extracted from the negative example region of the input image and the feature amount extracted from the positive example region of the output image, the larger the loss. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Taesung Park, Alexei A. Efros, Richard Zhang, Jun-Yan Zhu, “Contrastive Learning for Unpaired Image-to-Image Translation”, arXiv, arXiv:2007.15651, August 20, 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] Non-Patent Document 1 does not mention the influence on training of which region of an input image is extracted as a positive example region. The present disclosure has been made in consideration of the above problems, and one of its purposes is to provide a new technique for training a model that converts images. [Means for solving the problem]

[0005] The first model training device of the present disclosure has a training execution unit that trains an image conversion model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment, and an acquisition unit that acquires a training image, which is a first image used for the training. The image transformation model includes a feature extraction model that extracts features from each of a plurality of subregions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model. The training execution unit inputs the training image to the image conversion model to obtain a first feature map that is a feature map of the training image and the output image, and inputs the output image to the feature extraction model to obtain a second feature map that is a feature map of the output image, generates one or more patch sets that are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region in the output image at a position corresponding to the first positive example patch, and trains the image conversion model using a patch-wise loss calculated using the first feature map, the second feature map, and the patch set. In generating the patch set, the training execution unit extracts the first positive example patches from specific regions that represent a specific type of object among regions included in the training image.

[0006] A first control method of the present disclosure is executed by a computer and includes a training step of training an image transformation model to generate an output image representing a scene in a second environment from an input image representing a scene in a first environment, and an acquisition step of acquiring a training image, which is a first image used in the training. The image transformation model includes a feature extraction model that extracts features from each of a plurality of subregions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model. In the training execution step, the training image is input into the image conversion model to obtain a first feature map that is a feature map of the training image and the output image, and the output image is input into the feature extraction model to obtain a second feature map that is a feature map of the output image, and one or more patch sets are generated that are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region in the output image at a position corresponding to the first positive example patch, and the image conversion model is trained using a patch-wise loss calculated using the first feature map, the second feature map, and the patch set. In the training execution step, in generating the patch set, the first positive example patches are extracted with a focus on a specific region that represents a specific type of object among regions included in the training image.

[0007] A first computer-readable medium of the present disclosure stores a program for causing a computer to execute the first control method of the present disclosure.

[0008] The second model training device of the present disclosure has a training execution unit that trains an image conversion model to generate an output image representing a scene in a second environment from an input image representing a scene in a first environment, and an acquisition unit that acquires a training image, which is a first image used for the training. The image transformation model includes a feature extraction model that extracts features from each of a plurality of subregions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model. The training execution unit inputs the training image into the image conversion model to obtain a first feature map that is a feature map of the training image and the output image, and inputs the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image, extracts a plurality of patch sets that are a set of a first positive example patch and a first negative example patch that are partial areas of the training image, and a second positive example patch that is an area at a position in the output image corresponding to the first positive example patch, and for each of the patch sets, extracts a feature set that is a set of a first positive example patch and a first negative example patch that are partial areas of the training image, and a second positive example patch that is an area at a position in the output image corresponding to the first positive example patch, and extracts a feature set that is a set of a first positive example patch and a first negative example patch that are partial areas of the training image, a loss is calculated so that the magnitude of the loss becomes smaller as the similarity between the feature amount corresponding to the first negative example patch indicated by the first feature map and the feature amount corresponding to the second positive example patch indicated by the second feature map increases, a patch-wise loss is calculated using the loss calculated for each of the patch sets so that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific area representing a specific type of object is larger than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific area, and the image transformation model is trained using the patch-wise loss.

[0009] A second control method of the present disclosure is executed by a computer and includes a training step of training an image transformation model to generate an output image representing a scene in a second environment from an input image representing a scene in a first environment, and an acquisition step of acquiring a training image, which is a first image used in the training. The image transformation model includes a feature extraction model that extracts features from each of a plurality of subregions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model. In the training execution step, the training image is input to the image conversion model to obtain a first feature map which is a feature map of the training image and the output image, and the output image is input to the feature extraction model to obtain a second feature map which is a feature map of the output image, and a plurality of patch sets which are a set of a first positive example patch and a first negative example patch which are partial regions of the training image, and a second positive example patch which is a region in the output image at a position corresponding to the first positive example patch are extracted, and for each of the patch sets, a similarity between a feature corresponding to the first positive example patch indicated by the first feature map and a feature corresponding to the second positive example patch indicated by the second feature map is calculated. a loss is calculated so that the magnitude of the loss decreases as the similarity between the feature amount corresponding to the first negative example patch indicated by the first feature map and the feature amount corresponding to the second positive example patch indicated by the second feature map increases, and a patchwise loss is calculated using the loss calculated for each of the patch sets so that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific area representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific area, and the image transformation model is trained using the patchwise loss.

[0010] A second computer-readable medium of the present disclosure stores a program for causing a computer to execute the second control method of the present disclosure. Effect of the Invention

[0011] In accordance with the present disclosure, new techniques are provided for training models for performing image transformations. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 illustrates an example structure of an image transformation model trained by the model training device. [Diagram 2] FIG. 2 is a diagram illustrating an example of an outline of the operation of the model training device. [Diagram 3] FIG. 1 is a diagram for explaining an overview of patchwise loss. [Figure 4] FIG. 2 is a block diagram illustrating a functional configuration of a model training device. [Diagram 5] FIG. 2 is a block diagram illustrating a hardware configuration of a computer that realizes the model training device. [Figure 6] 11 is a flowchart illustrating a process executed by the model training device. [Figure 7] 11A and 11B are diagrams illustrating the effect of data augmentation using an image transformation model. [Figure 8] 1 is a diagram illustrating an example of a relationship between a partial region on an image and a feature map obtained from the image. [Figure 9] FIG. 1 illustrates an example of a generative adversarial network composed of an image transformation model and a discriminative model. [Figure 10] FIG. 13 is a diagram illustrating an example of an extraction probability map. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicated explanations are omitted as necessary for clarity of explanation. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or more arbitrary number of storage devices.

[0014] [Embodiment 1] <Summary> 1 is a diagram illustrating the structure of an image transformation model 100 trained by the model training device of embodiment 1. The image transformation model 100 outputs an output image 20 in response to an input of an input image 10. The input image 10 is an image input to the image transformation model 100. The output image 20 is an image output from the image transformation model 100.

[0015] The image conversion model 100 is trained to perform the process of "when an image showing a scene in a first environment is input as the input image 10, output an image showing the scene in a second environment different from the first environment as the output image 20." This enables the image conversion model 100 to generate a pseudo image of a scene captured in a certain environment from an image of the scene captured in another environment.

[0016] For example, the first environment is assumed to be daytime and the second environment is assumed to be nighttime. Also, the input image 10 is assumed to be an image obtained by capturing an image of a specific road with a camera. Here, the appearance of a road at night differs from the appearance of a road during the day in that the road is dark overall, various lights such as car lights and street lights are on, and places illuminated by lights are brighter than other places. The image conversion model 100 generates an image of a road at night from an image of a road at daytime so as to reproduce the characteristics of such a road at night in a pseudo manner. This makes it possible to realize data augmentation, for example, as described later.

[0017] The environment is not limited to a time period such as daytime or nighttime. Another example of the environment is an environment related to weather. For example, assume that a first environment is sunny and a second environment is rainy. In this case, the image conversion model 100 generates an output image 20 representing a scene under rainy weather from an input image 10 representing a scene under sunny weather. Instead of rain, weather such as snow can be used.

[0018] The image conversion model 100 includes two models, a feature extraction model 110 and an image generation model 120. The feature extraction model 110 is a model trained to extract a feature map from an input image. Here, the feature map extracted from the image is a set of features obtained from each of multiple partial regions included in the image. The image generation model 120 is a model trained to generate an output image 20 from the feature map.

[0019] Both the feature extraction model 110 and the image generation model 120 may be configured as any type of machine learning model, for example, both the feature extraction model 110 and the image generation model 120 may be configured as a neural network.

[0020] Fig. 2 is a diagram illustrating an example of an outline of the operation of the model training device 2000 of embodiment 1. Here, Fig. 2 is a diagram for facilitating understanding of the outline of the model training device 2000, and the operation of the model training device 2000 is not limited to that shown in Fig. 2.

[0021] To train the image conversion model 100, the model training device 2000 acquires a plurality of training images 50 representing a scene captured under a first environment. The model training device 2000 performs the following process using each training image 50. First, the model training device 2000 inputs the training image 50 to the image conversion model 100 to obtain a first feature map 30, which is a feature map of the training image 50, and an output image 20. Furthermore, the model training device 2000 inputs the output image 20 to the feature extraction model 110 to obtain a second feature map 40, which is a feature map of the output image 20. The model training device 2000 calculates a loss using the first feature map 30 and the second feature map 40, and uses the calculated loss to train the image conversion model 100. Hereinafter, the loss calculated using the first feature map 30 and the second feature map 40 is referred to as a patchwise loss. As described later, in addition to the patchwise loss, other losses may be used to train the image conversion model 100.

[0022] The patch-wise loss will be outlined with reference to FIG. 3. FIG. 3 is a diagram for explaining the outline of the patch-wise loss. The model training device 2000 extracts a positive example patch 52 and a negative example patch 54 from a training image 50. Both the positive example patch 52 and the negative example patch 54 are part of an image region of the training image 50. The model training device 2000 also extracts a positive example patch 22 from the output image 20. The positive example patch 22 is a region that represents the same location as the location represented by the positive example patch 52, among the part of the image region of the output image 20. Hereinafter, a combination of the positive example patch 52, the negative example patch 54, and the positive example patch 22 is referred to as a patch set. The model training device 2000 generates one or more patch sets.

[0023] Here, in the image input to the image conversion model 100 and the image output from the image conversion model 100, a scene in a first environment is converted to a scene in a second environment, but the degree of similarity between image regions that should represent the same place in these images should be greater than the degree of similarity between regions that should represent different places. Therefore, the model training device 2000 reduces the magnitude of the patchwise loss as the similarity between the feature corresponding to the positive example patch 52 among the features included in the first feature map 30 and the feature corresponding to the positive example patch 22 among the features included in the second feature map 40 increases. In addition, the model training device 2000 increases the magnitude of the patchwise loss as the similarity between the feature corresponding to the negative example patch 54 among the features included in the first feature map 30 and the feature corresponding to the positive example patch 22 among the features included in the second feature map 40 increases.

[0024] Here, the model training device 2000 extracts positive example patches 52 primarily from partial regions (hereinafter, specific regions) that represent a specific type of object from the image regions of the training image 50. Here, "primarily" means that the number of cases in which positive example patches 52 are extracted from the specific regions is greater than the number of cases in which positive example patches 52 are extracted from other partial regions. Therefore, for example, the positive example patches 52 are extracted such that the probability of extracting positive example patches 52 from the specific region is higher than the probability of extracting positive example patches 52 from other partial regions.

[0025] By thus focusing on extracting the positive example patches 52 from a specific region, the feature extraction model 110 can be trained so that the features of a specific type of object are extracted with particularly high accuracy from among the features of the scene represented by the input image 10. Thus, the image conversion model 100 can be trained so that the conversion of the feature of interest is performed with particularly high accuracy.

[0026] For example, as described above, the image conversion model 100 is trained to generate an output image 20 representing a nighttime road scene from an input image 10 representing the daytime road scene. Then, it is assumed that an object of particular interest in the input image 10 and the output image 20 is a car. In this case, when extracting the positive example patch 52 from the training image 50, the model training device 2000 extracts the positive example patch 52 from a partial region representing a car among the image regions included in the training image 50. In this way, the feature extraction model 110 is trained so that the features of the car are extracted with particularly high accuracy for the scene represented by the input image 10. Therefore, in the process of the image conversion model 100 of converting an image representing a daytime road scene into an image representing the corresponding scene at night, the image conversion model 100 can be trained so that the conversion from a daytime car to a nighttime car is performed with particularly high accuracy. In this way, it is possible to obtain an image conversion model 100 that can perform the conversion from a daytime car to a nighttime car with high accuracy.

[0027] The model training device 2000 of this embodiment will be described in more detail below.

[0028] <Example of functional configuration> FIG. 4 is a block diagram illustrating a functional configuration of the model training device 2000 of the first embodiment. The model training device 2000 includes an acquisition unit 2020 and a training execution unit 2040. The acquisition unit 2020 acquires a plurality of training images 50. The training execution unit 2040 trains the image conversion model 100 using the training images 50. Specifically, the training execution unit 2040 inputs the training images 50 to the image conversion model 100 to obtain a first feature map 30 and an output image 20. The training execution unit 2040 also inputs the output image 20 to the feature extraction model 110 to obtain a second feature map 40. The training execution unit 2040 calculates a patchwise loss using the first feature map 30 and the second feature map 40, and updates the parameters of the image conversion model 100 using the patchwise loss.

[0029] Here, the training execution unit 2040 extracts positive example patches 52 and negative example patches 54 from the training image 50. The training execution unit 2040 also extracts positive example patches 22 from the output image 20. The positive example patches 52 are extracted with a focus on specific regions that represent specific types of objects among partial regions included in the training image 50.

[0030] The patchwise loss is calculated so that the magnitude is smaller as the similarity between a feature amount corresponding to the positive example patch 52 among the features included in the first feature map 30 and a feature amount corresponding to the positive example patch 22 among the features included in the second feature map 40 is higher. The patchwise loss is calculated so that the magnitude is larger as the similarity between a feature amount corresponding to the negative example patch 54 among the features included in the first feature map 30 and a feature amount corresponding to the positive example patch 22 among the features included in the second feature map 40 is higher.

[0031] <Example of hardware configuration> Each functional component of the model training device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the model training device 2000 is realized by a combination of hardware and software.

[0032] 5 is a block diagram illustrating an example of a hardware configuration of a computer 500 that realizes the model training device 2000. The computer 500 is any computer. For example, the computer 500 is a stationary computer such as a PC (Personal Computer) or a server machine. In addition, for example, the computer 500 is a portable computer such as a smartphone or a tablet terminal. The computer 500 may be a dedicated computer designed to realize the model training device 2000, or may be a general-purpose computer.

[0033] For example, by installing a specific application on computer 500, each function of model training device 2000 is realized by computer 500. The application is configured with a program for realizing each functional component of model training device 2000. The method of acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD disk or a USB memory) in which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device in which the program is stored.

[0034] The computer 500 has a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to transmit and receive data to and from each other. However, the method of connecting the processor 504 and other components to each other is not limited to a bus connection.

[0035] The processor 504 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 506 is a main storage device realized using a random access memory (RAM) or the like. The storage device 508 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0036] The input / output interface 510 is an interface for connecting the computer 500 to an input / output device. For example, the input / output interface 510 is connected to an input device such as a keyboard and an output device such as a display device.

[0037] The network interface 512 is an interface for connecting the computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0038] The storage device 508 stores a program (a program for realizing the above-mentioned application) for realizing each functional component of the model training device 2000. The processor 504 reads this program into the memory 506 and executes it to realize each functional component of the model training device 2000.

[0039] The model training device 2000 may be realized by one computer 500, or may be realized by multiple computers 500. In the latter case, the configurations of the computers 500 do not need to be the same, and may be different from each other.

[0040] <Processing flow> FIG. 6 is a flowchart illustrating a process flow executed by the model training device 2000 of the first embodiment. The acquisition unit 2020 acquires a plurality of training images 50 (S102). Steps S104 to S116 constitute a loop process L1 that is repeatedly executed until a predetermined end condition is satisfied. In S104, the training execution unit 2040 determines whether the end condition is satisfied. If the end condition is satisfied, the training execution unit 2040 outputs the processing result (S118). On the other hand, if the end condition is not satisfied, the training execution unit 2040 selects one of the training images 50 that have not yet been the target of the loop process L1. The training image 50 selected here is denoted as a training image i.

[0041] The training execution unit 2040 inputs the training image i to the image conversion model 100 to obtain the first feature map 30 and the output image 20 (S106). The training execution unit 2040 inputs the output image 20 to the feature extraction model 110 to obtain the second feature map 40 (S108). The training execution unit 2040 generates one or more patch sets that are combinations of the positive example patches 52, the negative example patches 54, and the positive example patches 22 (S110). The training execution unit 2040 calculates a patch-wise loss based on the first feature map 30, the second feature map 40, and the patch set (S112). The training execution unit 2040 updates the image conversion model 100 using the patch-wise loss (S114). Since S116 is the end of the loop process L1, the process of FIG. 4 proceeds to S104.

[0042] Here, various conditions can be adopted as the termination condition of the loop process L1, that is, the termination condition of the training of the image conversion model 100. For example, the termination condition is a condition that "the loop process L1 is executed for all acquired training images 50". Another example of the termination condition is a condition that "the training of the image conversion model 100 converges". Note that various existing technologies can be used as a technology for determining whether the training of the machine learning model has converged.

[0043] The process flow shown in FIG. 6 is an example, and the process flow executed by the model training device 2000 is not limited to the process flow shown in FIG. 6. For example, instead of calculating the patchwise loss for each training image 50, the patchwise loss may be calculated for multiple training images 50 collectively. In this case, at the beginning of the loop process L1, multiple training images 50 are selected instead of one. Furthermore, steps S106 to S110 are performed for each of the multiple training images 50. Then, in S112, the patchwise loss is calculated using the first feature map 30, the second feature map 40, and the patch set obtained from each of the multiple training images 50.

[0044] <Example of using Image Conversion Model 100> To facilitate understanding of the usefulness of the model training device 2000, an example of a usage scene of the image transformation model 100 will be described. The usage scene described here is merely an example, and the usage scene of the model training device 2000 is not limited to the example described below.

[0045] As a usage scenario, a case is assumed in which video data obtained from a surveillance camera capturing images of a road is used to monitor vehicles. The surveillance device detects vehicles from each video frame of the video data using a surveillance device. The surveillance device has a detection model that is pre-trained to detect vehicles from images.

[0046] Here, the appearance of an object in an image (image features of the object) may differ depending on the environment in which the object is captured. For example, a car captured during the day and a car captured at night will look different from each other. Also, a car captured on a sunny day and a car captured on a rainy day will look different from each other.

[0047] It is preferable that a detection model used for vehicle monitoring is robust to such environmental changes. That is, the detection model needs to be trained so that it can detect vehicles from each video frame regardless of the time of day or weather. To achieve this, it is necessary to train the detection model using images of roads captured under various environments as training images.

[0048] In this regard, the ease of obtaining training images may differ depending on the environment. For example, since there are fewer cars at night compared to daytime, fewer images of cars on roads at night can be obtained from a surveillance camera compared to images of cars on roads during the day. Also, in places where it is mostly sunny, fewer images of cars on roads other than sunny days, such as rain and snow, can be obtained from a surveillance camera compared to images of cars on roads during sunny days. Due to the difference in the number of images available in this way depending on the environment, if a detection model is trained using only images available from a surveillance camera, the detection accuracy of cars in environments such as nighttime and rainy weather will be low.

[0049] Therefore, by using the image transformation model 100 trained by the model training device 2000, data augmentation is performed based on images of environments that are easy to obtain, and images of environments that are difficult to obtain are generated in a pseudo manner. For example, assume that the image transformation model 100 has been trained in advance by the model training device 2000 so that an image of a car on a road at night is output as the output image 20 in response to an image of a car on a road during the day being input as the input image 10.

[0050] Fig. 7 is a diagram illustrating the effect of data augmentation using the image conversion model 100. The upper part of Fig. 7 shows a case where the detection model is trained using only images obtained from a surveillance camera without data augmentation using the image conversion model 100. In this case, the number of training images in which cars are captured at night is insufficient, resulting in low detection accuracy for cars at night.

[0051] On the other hand, the lower part of FIG. 7 illustrates a case where data expansion is performed by the image conversion model 100. A user inputs an image of a car on a road during the day obtained from a surveillance camera into the image conversion model 100 to obtain an image that simulates a car on a road at night. In this way, it is possible to obtain the same number of images of cars on roads at night as the number of images of cars on roads during the day. In this way, by training the detection model using the images obtained using the image conversion model 100 as training images, it is possible to generate a detection model that can accurately detect cars at night. In other words, it is possible to generate a detection model that is robust to changes in the environment.

[0052] <Acquisition of training image 50: S102> The acquisition unit 2020 acquires a plurality of training images 50 (S102). There are various methods for the acquisition unit 2020 to acquire the training images 50. For example, the training images 50 are stored in advance in an arbitrary storage device in a form that can be acquired from the model training device 2000. In this case, the acquisition unit 2020 reads the training images 50 from the storage device. Alternatively, for example, the acquisition unit 2020 may acquire the training images 50 by receiving the training images 50 transmitted from another device.

[0053] <Loss calculation method: S112> The training execution unit 2040 calculates a patch-wise loss based on the first feature map 30, the second feature map 40, and the patch set. The first feature map 30 includes features for each of a plurality of regions included in the input image 10. The training execution unit 2040 extracts features corresponding to the positive example patch 52 and features corresponding to the negative example patch 54 from the first feature map 30. Similarly, the second feature map 40 includes features for each of a plurality of regions included in the output image 20. The training execution unit 2040 extracts features corresponding to the positive example patch 22 from the second feature map 40. The training execution unit 2040 then calculates a patch-wise loss using the features corresponding to the positive example patch 52, the features corresponding to the negative example patch 54, and the features corresponding to the positive example patch 22.

[0054] <<Feature extraction method>> As a specific method for obtaining feature amounts for a specific partial region of an input image from a feature map, an existing method can be used. This method will be briefly described below. FIG. 8 is a diagram illustrating an example of the relationship between a partial region on an image and a feature map obtained from the image. In this example, the feature extraction model 110 is configured to perform a filter convolution operation on a plurality of partial regions in the input image 10.

[0055] A cell 210 of the feature map 200 indicates a feature obtained by performing a convolution operation of a filter on a subregion 12 of the input image 10. Therefore, a feature corresponding to the entire subregion 12 or an area inside the subregion 12 can be obtained from the cell 210 of the feature map 200.

[0056] 8, it is assumed that the input image 10 is a training image 50, the partial region 12 is a positive example patch 52, and the feature map 200 is the first feature map 30. In this case, the feature amount corresponding to the positive example patch 52 can be obtained from a cell in the first feature map 30 that corresponds to the cell 210.

[0057] It is possible to identify which cell of the first feature map 30 contains the feature amount corresponding to the positive example patch 52 from the position of the positive example patch 52 in the training image 50. The training execution unit 2040 then identifies the cell of the first feature map 30 that corresponds to the positive example patch 52, and extracts the feature amount corresponding to the positive example patch 52 from the identified cell.

[0058] Similarly, it is possible to identify which cell of the first feature map 30 contains the feature amount corresponding to the negative example patch 54 from the position of the negative example patch 54 in the training image 50. The training execution unit 2040 then identifies the cell of the first feature map 30 that corresponds to the negative example patch 54, and extracts the feature amount corresponding to the negative example patch 54 from the identified cell.

[0059] Similarly, it is possible to identify which cell of the second feature map 40 contains the feature amount corresponding to the positive example patch 22 from the position of the positive example patch 22 in the output image 20. The training execution unit 2040 then identifies the cell of the second feature map 40 that corresponds to the positive example patch 22, and extracts the feature amount corresponding to the positive example patch 22 from the identified cell.

[0060] The feature extraction model 110 may be configured to extract multiple types of feature amounts from each partial region of the input image 10. For example, the feature extraction model 110 is configured to perform a convolution operation for each of multiple filters on the input image 10. In this case, the feature amount of each partial region of the input image 10 is obtained for each filter. Thus, the two-dimensional map illustrated in FIG. 8 is obtained for each filter. The training execution unit 2040 then extracts feature amounts such as the positive example patch 52 from each map.

[0061] <<How to calculate patchwise loss from features>> The training execution unit 2040 extracts a feature corresponding to the positive example patch 52, a feature corresponding to the negative example patch 54, and a feature corresponding to the positive example patch 22 for each patch set. The training execution unit 2040 then calculates a patchwise loss using these feature amounts extracted for each patch set. For the patchwise loss, for example, the patchwise contrastive loss disclosed in Non-Patent Document 1 can be used. A specific example of a method for calculating the patchwise loss will be described below.

[0062] The loss for one patch set is calculated, for example, as the cross-entropy loss expressed by the following equation (1).

number

[0063] When there is one patch set, the patchwise loss is calculated by the above formula (1). On the other hand, when there are multiple patches, the patchwise loss can be generalized as the following formula (2).

number

[0064] The feature extraction model 110 may be configured to perform multi-stage feature extraction. For example, an example of such a feature extraction model 110 is a convolutional neural network having multiple convolutional layers. In a convolutional neural network having multiple convolutional layers, the n-th convolutional layer outputs the n-th feature map (n is an integer of 2 or more) by performing a convolution operation of the (n-1)th filter on the (n-1)th feature map output from the (n-1)th convolutional layer.

[0065] When multi-stage feature extraction is performed in this way, not only the first feature map 30 and the second feature map 40, which are the final feature maps, but also feature maps obtained in intermediate stages can be used to calculate the patchwise loss. That is, a plurality of feature maps obtained from the training images 50 and a plurality of feature maps obtained from the output image 20 can be used to calculate the patchwise loss.

[0066] For example, when the feature extraction model 110 is an n-layer convolutional neural network, n feature maps can be obtained by obtaining feature maps from each layer. Then, features corresponding to the positive example patch 52, the negative example patch 54, and the positive example patch 22 can be extracted from each of the n feature maps. The training execution unit 2040 extracts features corresponding to the positive example patch 52, the negative example patch 54, and the positive example patch 22 from each of the n feature maps, and calculates the patch-wise loss using the extracted features.

[0067] When calculating the patch-wise loss using multiple feature maps obtained from the training images 50 and the output image 20, the patch-wise loss is expressed by, for example, the following equation (3).

number

[0068] Also, as described above, the patch-wise loss may be calculated collectively for multiple training images 50. In this case, the patch-wise loss can be generalized by the following equation (4):

number

[0069] <<Other losses>> In addition to the patchwise loss, it is preferable to further use the adversarial loss described below for training the image conversion model 100. In this case, a generative adversarial network (GAN) is configured using the image conversion model 100 and a discriminative model as a premise. FIG. 9 is a diagram illustrating a generative adversarial network configured with the image conversion model 100 and a discriminative model.

[0070] The generative adversarial network 70 is composed of an image transformation model 100 and a discrimination model 80. The discrimination model 80 is trained to output a label 90 indicating whether an image is an image generated by the image transformation model 100 (i.e., the output image 20) or not, in response to an image being input. Therefore, when an image obtained by capturing an image of a scene in a second environment with a camera is input, the trained discrimination model 80 outputs a label 90 indicating that the image is not the output image 20. In addition, when the output image 20 generated by the image transformation model 100 is input, the trained discrimination model 80 outputs a label 90 indicating that the image is the output image 20.

[0071] The training execution unit 2040 obtains an output image 20 by inputting a training image 50 to the image transformation model 100, and then inputs the output image 20 to the discriminative model 80. The training execution unit 2040 then calculates the adversarial loss using a label 90 obtained from the discriminative model 80. Here, an existing method can be used as a specific method for calculating the adversarial loss in the generative adversarial network.

[0072] The training execution unit 2040 uses the patchwise loss and the adversarial loss to train the image conversion model 100. For example, the training execution unit 2040 calculates the sum or weighted sum of the patchwise loss and the adversarial loss as an overall loss, and uses the overall loss to train the generative adversarial network 70 (training the feature extraction model 110, the image generation model 120, and the discriminative model 80). Note that the overall loss may further include types of losses other than the patchwise loss and the adversarial loss.

[0073] Training a model using a loss is performed by updating the parameters that make up the model based on the loss. Here, existing methods can be used as a specific method for updating the parameters of each model that makes up the generative adversarial network using the loss.

[0074] <Patch set generation: S110> The training execution unit 2040 generates patch sets for the training image 50 and the output image 20. As described above, one patch set includes one positive example patch 52, one or more negative example patches 54, and one positive example patch 22. For example, the training execution unit 2040 performs a process of extracting the positive example patch 52 from the training image 50, and then performs a process of extracting one or more negative example patches 54 from an area of ​​the training image 50 other than the positive example patch 52, and a process of extracting the positive example patch 22 from the output image 20.

[0075] The positive example patches 52 are extracted primarily from specific regions, which are partial regions of the training images 50 that contain a specific type of object. Therefore, the training execution unit 2040 detects specific regions from the training images 50 to be used for extracting the positive example patches 52. Here, existing technology can be used as a technology for detecting regions representing a specific type of object from the training images 50.

[0076] The specific type may be determined in advance or may be specified by the user. In the latter case, the training execution unit 2040 acquires information representing the specific type and detects the area representing the specific type indicated in the information as the specific area. The information representing the specific type is obtained as a result of user input, for example.

[0077] Below, several examples of methods for extracting the positive example patch 52 based on the detected specific region will be described.

[0078] <<Method 1>> First, the training execution unit 2040 determines whether to extract the positive example patches 52 from inside or outside the specific region. This determination is made so that the number of positive example patches 52 extracted from inside the specific region is greater than the number of positive example patches 52 extracted from outside the specific region. In this way, the positive example patches 52 are extracted with a focus on the specific region.

[0079] For example, the above-mentioned determination is made probabilistically. As a method for probabilistically selecting one of the two options in this way, for example, a method of sampling a value from a Bernoulli distribution and making a determination based on the sample value is conceivable. More specifically, for example, when the sample value is 1, a positive example patch 52 is extracted from within the specific region, and when the sample value is 0, a positive example patch 52 is extracted from outside the specific region. In this case, by making the probability that the sample value is 1 larger than 50%, the number of positive example patches 52 extracted from within the specific region can be probabilistically made larger than the number of positive example patches 52 extracted from outside the specific region.

[0080] After determining whether the positive example patch 52 is to be extracted from inside or outside the specific region, the training execution unit 2040 extracts the positive example patch 52 based on the determination. Here, the size of the positive example patch 52 (hereinafter, patch size) is determined in advance. When extracting the positive example patch 52 from inside the specific region, the training execution unit 2040 extracts a patch-sized area from an arbitrary location within the specific region, and treats this area as the positive example patch 52. On the other hand, when extracting the positive example patch 52 from outside the specific region, the training execution unit 2040 selects a patch-sized area from an arbitrary location outside the specific region, and determines the selected area as the positive example patch 52. Note that an existing technology can be used as a technology for arbitrarily selecting an area of ​​a predetermined size from a certain area.

[0081] In addition, when extracting the positive example patches 52 from within the specific region, a part of the positive example patches 52 may be outside the specific region. In this case, for example, the positive example patches 52 are extracted so as to satisfy the condition that "a predetermined ratio or more of the positive example patches 52 is within the specific region."

[0082] <<Method 2>> The training execution unit 2040 extracts the positive example patch 52 so that the greater the overlap with the specific region, the higher the probability of the region being extracted as the positive example patch 52. For this purpose, for example, the training execution unit 2040 generates an extraction probability map that indicates a higher extraction probability as the overlap rate with the specific region increases. For example, the extraction probability map is generated as a probability distribution that indicates the probability that a patch-sized region starting from that pixel (for example, the upper left corner of the positive example patch 52) is extracted as the positive example patch 52 for each pixel of the training image 50. In order to increase the extraction probability as the overlap rate with the specific region increases, the extraction probability map is generated so that the greater the overlap rate between the patch-sized region starting from that pixel and the specific region, the higher the extraction probability. It can also be said that the extraction probability map indicates the probability that each patch-sized partial region included in the training image 50 is extracted as the positive example patch 52. The extraction probability of each partial region is set higher as the overlap rate between the partial region and the specific region increases.

[0083] To generate such an extraction probability map, for example, the training execution unit 2040 first sets a value representing the degree of overlap between the patch-sized area starting from that pixel and the specific area for each pixel of the extraction probability map, and then changes the value of each pixel of the extraction probability map to a value obtained by dividing the value by the sum of the values ​​of all pixels.

[0084] FIG. 10 is a diagram illustrating an example of an extraction probability map. In this example, the size of the positive example patch 52 is 2x2. The size of the specific region 310 is 4x3. When a positive example patch 52 is extracted with the pixel at the upper left end, the greater the degree of overlap between the positive example patch 52 and the specific region, the higher the extraction probability of each pixel in the extraction probability map 300. Here, in FIG. 10, pixels with a higher extraction probability are represented by darker dots. Therefore, in FIG. 10, pixels represented by darker dots indicate a higher probability that the positive example patch 52 will be extracted with the pixel as the starting point.

[0085] The training execution unit 2040 samples pixel coordinates from the probability distribution represented by the extraction probability map, and extracts a patch-sized area starting from the sampled coordinates as a positive example patch 52.

[0086] <<Method 3>> The specific type of object may be further classified into smaller sub-classifications, and the extraction probability of each pixel in the above-mentioned extraction probability map may be determined based on the sub-classifications. For example, if the specific type is a car, the sub-classifications may be set to types such as passenger cars, trucks, and buses. Hereinafter, the type of the sub-classification to which the object included in the training image 50 belongs is referred to as a sub-classification.

[0087] When considering small classification, each sub-classification of objects belonging to a specific class may have a different importance in training the image conversion model 100. For example, objects of a class that appear infrequently in the training images 50 are considered to be important objects in training because it is necessary for the image conversion model 100 to learn the features of the objects with less training.

[0088] As a specific example, assume that an image showing the state of a road during the day is used as an input image 10, and the image conversion model 100 is trained to generate an output image 20 showing the state of a road at night from the input image 10. Here, assume that trucks appear less frequently than passenger cars on a road captured to generate a training image 50. In this case, there are fewer opportunities to learn the characteristics of trucks than to learn the characteristics of passenger cars. Therefore, it is necessary to enable the image conversion model 100 to learn the characteristics of trucks with less training.

[0089] Therefore, for example, the lower the frequency of occurrence of a sub-type, the higher the importance in training is given to that sub-type. More specifically, the training execution unit 2040 generates an extraction probability map such that the specific region representing an object belonging to a sub-type with a lower frequency of occurrence in the training image 50 has a higher extraction probability. For this purpose, a higher weight is set for each sub-type as its frequency of occurrence decreases.

[0090] The training execution unit 2040 sets a value for each pixel in the extraction probability map, which is the degree of overlap between the pixel and the specific region multiplied by a weight corresponding to the sub-type of the object represented by the specific region, and then changes the value of each pixel to a value divided by the sum of the values ​​of all pixels.

[0091] The training execution unit 2040 samples pixel coordinates from the probability distribution represented by this extraction probability map, and extracts a patch-sized area starting from the sampled coordinates as a positive example patch 52.

[0092] Here, the weight of each sub-type may be determined in advance or may be determined by the training execution unit 2040. In the latter case, for example, the training execution unit 2040 performs a process of detecting a specific type of object for each training image 50 acquired by the acquisition unit 2020 before extracting the positive example patch 52, and counts the number of detected objects for each sub-type. This identifies the number of occurrences of each sub-type in the training image group. The training execution unit 2040 determines a weight for each sub-type based on the number of occurrences of each sub-type. This weight is determined so that the weight of a sub-type that appears less frequently is larger.

[0093] <<How to extract negative example patch 54>> The training execution unit 2040 extracts an area of ​​a patch size arbitrarily (for example, randomly) from an area included in the training image 50 other than the area extracted as the positive example patch 52, and uses the area as a negative example patch 54. As described above, one patch set may include multiple negative example patches 54. The number of negative example patches 54 included in one patch set is determined in advance.

[0094] <<How to extract positive example patch 22>> The training execution unit 2040 extracts the positive example patch 22 from a position in the output image 20 that corresponds to the position on the training image 50 from which the positive example patch 52 was extracted. In other words, the same coordinates as those used as the starting point for extracting the positive example patch 52 are used as the pixel coordinates used as the starting point for extracting the positive example patch 22.

[0095] <Processing result output: S118> The model training device 2000 outputs, as a processing result, information (hereinafter, output information) capable of identifying the trained image transformation model 100. The output information includes at least a parameter group of the image transformation model 100 obtained by training. In addition, the output information may include a program for realizing the image transformation model 100.

[0096] The output information may be output in any manner. For example, the model training device 2000 may store the output information in any storage unit. Alternatively, the model training device 2000 may transmit the output information to another device (for example, a device used to operate the image transformation model 100).

[0097] <Modification> In the above-described model training device 2000, the features of a specific type of object are learned with particularly high accuracy by focusing on extracting positive example patches 52 from a specific region. However, the method for learning the features of a specific type of object with high accuracy is not limited to the method of focusing on extracting positive example patches 52 from a specific region.

[0098] For example, in addition to or instead of extracting the positive example patches 52 from a specific region with a focus, the model training device 2000 calculates a patch-wise loss such that the influence of a loss (e.g., the above-mentioned cross-entropy loss) calculated using features corresponding to the positive example patches 52 extracted from a specific region is greater than the influence of a loss calculated using features corresponding to the positive example patches 52 extracted from other regions. Note that, when the method of extracting the positive example patches 52 from a specific region with a focus is not adopted, for example, the positive example patches 52 are extracted with the same probability from any position of the training image 50.

[0099] Hereinafter, a method for determining the degree of influence of loss based on the feature amount corresponding to the positive example patch 52 depending on whether the positive example patch 52 is extracted from inside or outside the specific region will be described.

[0100] For example, the training execution unit 2040 calculates the patch-wise loss using the following equation (5).

number

[0101] In formula (5), the loss calculated for each patch set is multiplied by a weight α when the positive example patch 52 included in the patch set is extracted from within a specific region, and is multiplied by a weight β when the positive example patch 52 included in the patch set is extracted from outside the specific region. Since α>β>0, the impact of the loss when the positive example patch 52 is extracted from within the specific region is larger than the impact of the loss when the positive example patch 52 is extracted from outside the specific region.

[0102] The same can be applied to the case where the patch-wise loss is calculated using the above-mentioned formulas (3) and (4). That is, when feature maps are obtained from multiple layers of the feature extraction model 110, the above-mentioned weighting is applied to the loss calculated for the feature maps obtained from each layer.

[0103] However, when a feature map is obtained from multiple layers, a weight may be set for each layer or a weight may be assigned only to a specific layer based on the relationship between the size of a partial region of the input image corresponding to one cell of the feature map and the patch size. This method will be described below.

[0104] When feature maps are obtained from multiple layers, the size of the subregion of the input image corresponding to one cell of the feature map differs for each feature map (layer). For example, assume that convolution processing of a filter of size 3x3 is performed in each layer. In this case, one cell of the first feature map corresponds to a subregion of size 3x3 in the input image. Also, one cell of the second feature map corresponds to a set of cells of size 3x3 in the first feature map. Therefore, one cell of the second feature map corresponds to a region of size 9x9 in the input image. For the same reason, one cell of the third feature map corresponds to a region of size 27x27 in the input image. In this way, the feature maps generated by the later layers correspond to larger subregions of the input image.

[0105] In this regard, among multiple feature maps generated from different layers for the training image 50, the feature map in which the size of a partial region of the training image 50 corresponding to one cell is closer to the patch size is considered to more accurately represent the features of the positive example patch 52. The same is true for the negative example patch 54 and the positive example patch 22.

[0106] Therefore, for example, the training execution unit 2040 calculates the patch-wise loss so that the closer the size of a partial region of the training image 50 corresponding to one cell is to the patch size, the greater the weight is given to the feature amount extracted from that feature map. The same applies to the positive example patch 22 and the negative example patch 54. In this case, for example, the patch-wise loss is calculated using the following formula (6).

number

[0107] Note that a weight greater than 1 may be assigned only to the layer l in which the difference between z_p and z_l is the smallest, and no weighting may be assigned to other layers, so that weighting may be assigned only to the layer in which the size of the partial region of the input image corresponding to the cell of the feature map is closest to the patch size. Also, a method of assigning a weight greater than 1 to only a predetermined number of top layers in ascending order of the difference between z_p and z_l may be adopted.

[0108] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0109] In the above example, the program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0110] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) a training execution unit that trains an image transformation model to generate an output image representing a scene in a second environment from an input image representing a scene in a first environment; An acquisition unit that acquires a training image that is a first image used for the training, the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, The training execution unit includes: inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Generate one or more patch sets, which are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region at a position corresponding to the first positive example patch in the output image; training the image transformation model using the first feature map, the second feature map, and a patch-wise loss calculated using the patch set; The training execution unit, in generating the patch set, extracts the first positive example patches from specific regions that represent a specific type of object among regions included in the training image. (Appendix 2) the training execution unit performs a process of determining whether the first positive example patch should be extracted from inside or outside the specific region such that a probability that the first positive example patch is extracted from inside the specific region is higher than a probability that the first positive example patch is extracted from outside the specific region, and extracts the first positive example patch from the training image based on a result of the determination, thereby focusing on extracting the first positive example patch from the specific region. (Appendix 3) the training execution unit generates an extraction probability map representing an extraction probability, which is a probability that each of a plurality of partial regions of the training image will be extracted as the first positive example patch, and extracts the first positive example patch from the training image according to the extraction probability represented by the extraction probability map, thereby extracting the first positive example patch from the specific region with a focus on the first positive example patch; The model training device according to claim 1, wherein the training execution unit sets, for each partial region in the extraction probability map, a higher extraction probability the higher the degree of overlap between the partial region and the specific region. (Appendix 4) The training execution unit includes: Identifying a sub-class representing a finer classification than the specific class for the object represented by the specific region included in the training image; The model training device according to claim 3, wherein in the extraction probability map, for each partial region, the extraction probability is set to be higher the higher the degree of overlap between the partial region and the specific region and the lower the occurrence frequency of the sub-type of the object represented by the specific region overlapping with the partial region. (Appendix 5) The model training device according to any one of Additions 1 to 4, wherein the training execution unit calculates the patch-wise loss such that the magnitude of the patch-wise loss is smaller as a similarity between a feature corresponding to the first positive example patch and a feature corresponding to the second positive example patch is higher, and the magnitude of the patch-wise loss is larger as a similarity between a feature corresponding to the first negative example patch and a feature corresponding to the second positive example patch is higher. (Appendix 6) 1. A computer-implemented control method comprising: a training step of training the image transformation model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment; and acquiring a training image, which is a first image used for the training. the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Generate one or more patch sets, which are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region at a position corresponding to the first positive example patch in the output image; training the image transformation model using the first feature map, the second feature map, and a patch-wise loss calculated using the patch set; A control method in which, in the training execution step, in generating the patch set, the first positive example patches are extracted with a focus on a specific region that represents a specific type of object among regions included in the training image. (Appendix 7) The control method according to claim 6, wherein in the training execution step, a process of determining whether the first positive example patch is to be extracted from inside or outside the specific region is performed such that a probability that the first positive example patch is extracted from inside the specific region is higher than a probability that the first positive example patch is extracted from outside the specific region, and the first positive example patch is extracted from the training image based on a result of the determination, thereby focusing on extracting the first positive example patch from the specific region. (Appendix 8) In the training execution step, an extraction probability map is generated for each of a plurality of partial regions of the training image, which indicates an extraction probability that the partial region is extracted as the first positive example patch, and the first positive example patch is extracted from the training image according to the extraction probability indicated by the extraction probability map, thereby extracting the first positive example patch from the specific region with a focus on the extraction; The control method according to claim 6, wherein in the training execution step, a higher extraction probability is set for each partial region in the extraction probability map as the degree of overlap between the partial region and the specific region increases. (Appendix 9) In the training execution step, Identifying a sub-class representing a finer classification than the specific class for the object represented by the specific region included in the training image; The control method according to claim 8, wherein in the extraction probability map, the extraction probability is set for each partial region such that the extraction probability is higher the higher the degree of overlap between the partial region and the specific region and the lower the occurrence frequency of the sub-type of the object represented by the specific region overlapping with the partial region. (Appendix 10) The control method according to any one of Additions 6 to 9, wherein in the training execution step, the patch-wise loss is calculated so that the patch-wise loss is smaller as a similarity between a feature corresponding to the first positive example patch and a feature corresponding to the second positive example patch is higher, and the patch-wise loss is larger as a similarity between a feature corresponding to the first negative example patch and a feature corresponding to the second positive example patch is higher. (Appendix 11) On the computer, a training step of training the image transformation model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment; An acquisition step of acquiring a training image, which is a first image used for the training, and a program for executing the acquisition step are stored. the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Generate one or more patch sets, which are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region at a position corresponding to the first positive example patch in the output image; training the image transformation model using the first feature map, the second feature map, and a patch-wise loss calculated using the patch set; A non-transitory computer-readable medium, wherein in the training execution step, in generating the patch set, the first positive example patch is extracted with a focus on a specific region that represents a specific type of object among regions included in the training image. (Appendix 12) the step of executing training includes performing a process of determining whether the first positive example patch is to be extracted from inside or outside the specific region such that a probability that the first positive example patch is extracted from inside the specific region is higher than a probability that the first positive example patch is extracted from outside the specific region, and the first positive example patch is extracted from the training image based on a result of the determination, thereby focusing on extracting the first positive example patch from the specific region. (Appendix 13) In the training execution step, an extraction probability map is generated for each of a plurality of partial regions of the training image, which indicates an extraction probability that the partial region is extracted as the first positive example patch, and the first positive example patch is extracted from the training image according to the extraction probability indicated by the extraction probability map, thereby extracting the first positive example patch from the specific region with a focus on the extraction; The computer-readable medium of claim 11, wherein in the training execution step, a higher extraction probability is set for each subregion in the extraction probability map as the degree of overlap between the subregion and the specific region increases. (Appendix 14) In the training execution step, Identifying a sub-class representing a finer classification than the specific class for the object represented by the specific region included in the training image; The computer-readable medium of claim 13, wherein in the extraction probability map, the extraction probability is set for each partial region such that the extraction probability is higher the higher the degree of overlap between the partial region and the specific region and the lower the occurrence frequency of the sub-type of the object represented by the specific region that overlaps with the partial region. (Appendix 15) 15. The computer-readable medium of claim 11, wherein in the training execution step, the patch-wise loss is calculated such that the magnitude of the patch-wise loss is smaller as the similarity between the feature amount corresponding to the first positive example patch and the feature amount corresponding to the second positive example patch is higher, and the magnitude of the patch-wise loss is larger as the similarity between the feature amount corresponding to the first negative example patch and the feature amount corresponding to the second positive example patch is higher. (Appendix 16) a training execution unit that trains an image transformation model to generate an output image representing a scene in a second environment from an input image representing a scene in a first environment; An acquisition unit that acquires a training image that is a first image used for the training, the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, The training execution unit includes: inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Extracting a plurality of patch sets, each of which is a set of a first positive example patch and a first negative example patch that are a part of the training image, and a second positive example patch that is a region in the output image at a position corresponding to the first positive example patch; calculate a loss for each of the patch sets such that the magnitude of the loss decreases as the similarity between a feature amount corresponding to the first positive example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases, and the magnitude of the loss increases as the similarity between a feature amount corresponding to the first negative example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases; Using the loss calculated for each of the patch sets, calculate a patch-wise loss such that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific region representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific region; A model training device that uses the patch-wise loss to train the image transformation model. (Appendix 17) The training execution unit calculates the patch-wise loss by calculating a weighted sum of the losses calculated for each patch set; 17. The model training apparatus of claim 16, wherein, in the weighted sum, a loss weight calculated for the patch set including the first positive example patch extracted from within the specific region is set to a value greater than a loss weight calculated for the patch set including the first positive example patch extracted from outside the specific region. (Appendix 18) the feature extraction model has a plurality of layers; Each layer generates a feature map smaller in size than the feature map generated by the previous layer by extracting features from the feature map generated by the previous layer; The training execution unit includes: extracting a feature amount corresponding to the first positive example patch and a feature amount corresponding to the first negative example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the training image is input; extracting feature amounts corresponding to the second positive example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the output image is input; Calculating a loss for each of the patch sets and each of the layers using features of the first positive example patch, the first negative example patch, and the second positive example patch of the patch set extracted from the feature map generated by the layer; Calculating the patch-wise loss by calculating a weighted sum of the losses calculated for each of the patch sets and for each of the layers; The model training apparatus of claim 17, wherein in the weighted sum, the closer a size of a subregion of the input image corresponding to one cell of the feature map is to the size of the first positive example patch, the larger a weight of a loss calculated using features extracted from the feature map is set. (Appendix 19) the feature extraction model has a plurality of layers; Each layer generates a feature map smaller in size than the feature map generated by the previous layer by extracting features from the feature map generated by the previous layer; The training execution unit includes: extracting a feature amount corresponding to the first positive example patch and a feature amount corresponding to the first negative example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the training image is input; extracting feature amounts corresponding to the second positive example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the output image is input; Calculating a loss for each of the patch sets and each of the layers using features of the first positive example patch, the first negative example patch, and the second positive example patch of the patch set extracted from the feature map generated by the layer; Calculating the patch-wise loss by calculating a weighted sum of the losses calculated for each of the patch sets and for each of the layers; 18. The model training apparatus of claim 17, wherein, in the weighted sum, for a feature map in which a size of a subregion corresponding to one cell of the feature map is closest to a size of the first positive example patch, a weight is set that is greater than that for other losses for a loss calculated using features extracted from that feature map. (Appendix 20) 1. A computer-implemented control method comprising: a training step of training the image transformation model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment; and acquiring a training image, which is a first image used for the training. the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Extracting a plurality of patch sets, each of which is a set of a first positive example patch and a first negative example patch that are a part of the training image, and a second positive example patch that is a region in the output image at a position corresponding to the first positive example patch; calculate a loss for each of the patch sets such that the magnitude of the loss decreases as the similarity between a feature amount corresponding to the first positive example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases, and the magnitude of the loss increases as the similarity between a feature amount corresponding to the first negative example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases; Using the loss calculated for each of the patch sets, calculate a patch-wise loss such that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific region representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific region; A control method, training the image transformation model using the patch-wise loss. (Appendix 21) In the training step, a weighted sum of the losses calculated for each patch set is calculated to calculate the patch-wise loss; 21. The control method of claim 20, wherein in the weighted sum, a loss weight calculated for the patch set including the first positive example patch extracted from within the specific region is set to a value greater than a loss weight calculated for the patch set including the first positive example patch extracted from outside the specific region. (Appendix 22) the feature extraction model has a plurality of layers; Each layer generates a feature map smaller in size than the feature map generated by the previous layer by extracting features from the feature map generated by the previous layer; In the training execution step, extracting a feature amount corresponding to the first positive example patch and a feature amount corresponding to the first negative example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the training image is input; extracting feature amounts corresponding to the second positive example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the output image is input; Calculating a loss for each of the patch sets and each of the layers using features of the first positive example patch, the first negative example patch, and the second positive example patch of the patch set extracted from the feature map generated by the layer; Calculating the patch-wise loss by calculating a weighted sum of the losses calculated for each of the patch sets and for each of the layers; The control method of claim 21, wherein in the weighted sum, the closer a size of a subregion corresponding to one cell of the feature map is to the size of the first positive example patch in a feature map, the larger a weight of the loss calculated using the features extracted from that feature map is set. (Appendix 23) the feature extraction model has a plurality of layers; Each layer generates a feature map smaller in size than the feature map generated by the previous layer by extracting features from the feature map generated by the previous layer; In the training execution step, extracting a feature amount corresponding to the first positive example patch and a feature amount corresponding to the first negative example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the training image is input; extracting feature amounts corresponding to the second positive example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the output image is input; Calculating a loss for each of the patch sets and each of the layers using features of the first positive example patch, the first negative example patch, and the second positive example patch of the patch set extracted from the feature map generated by the layer; Calculating the patch-wise loss by calculating a weighted sum of the losses calculated for each of the patch sets and for each of the layers; The control method of claim 21, wherein in the weighted sum, for a feature map in which a size of a subregion of the input image corresponding to one cell of the feature map is closest to a size of the first positive example patch, a weight is set that is greater than that for other losses for a loss calculated using features extracted from that feature map. (Appendix 24) On the computer, a training step of training the image transformation model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment; An acquisition step of acquiring a training image, which is a first image used for the training, and a program for executing the acquisition step are stored. the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Extracting a plurality of patch sets, each of which is a set of a first positive example patch and a first negative example patch that are a part of the training image, and a second positive example patch that is a region in the output image at a position corresponding to the first positive example patch; calculate a loss for each of the patch sets such that the magnitude of the loss decreases as the similarity between a feature amount corresponding to the first positive example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases, and the magnitude of the loss increases as the similarity between a feature amount corresponding to the first negative example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases; Using the loss calculated for each of the patch sets, calculate a patch-wise loss such that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific region representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific region; A non-transitory computer-readable medium for training the image transformation model using the patch-wise loss. (Appendix 25) In the training step, a weighted sum of the losses calculated for each patch set is calculated to calculate the patch-wise loss; 25. The computer-readable medium of claim 24, wherein in the weighted sum, a loss weight calculated for the patch set including the first positive example patch extracted from within the specific region is set to a value greater than a loss weight calculated for the patch set including the first positive example patch extracted from outside the specific region. (Appendix 26) the feature extraction model has a plurality of layers; Each layer generates a feature map smaller in size than the feature map generated by the previous layer by extracting features from the feature map generated by the previous layer; In the training execution step, extracting a feature amount corresponding to the first positive example patch and a feature amount corresponding to the first negative example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the training image is input; extracting feature amounts corresponding to the second positive example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the output image is input; Calculating a loss for each of the patch sets and each of the layers using features of the first positive example patch, the first negative example patch, and the second positive example patch of the patch set extracted from the feature map generated by the layer; Calculating the patch-wise loss by calculating a weighted sum of the losses calculated for each of the patch sets and for each of the layers; The computer-readable medium of claim 25, wherein in the weighted sum, the closer the size of a subregion corresponding to one cell of the feature map is to the size of the first positive example patch, the larger the weight of the loss calculated using the features extracted from that feature map is set. (Appendix 27) the feature extraction model has a plurality of layers; Each layer generates a feature map smaller in size than the feature map generated by the previous layer by extracting features from the feature map generated by the previous layer; In the training execution step, extracting a feature amount corresponding to the first positive example patch and a feature amount corresponding to the first negative example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the training image is input; extracting feature amounts corresponding to the second positive example patch from each of a plurality of feature maps generated by the plurality of layers of the feature extraction model to which the output image is input; Calculating a loss for each of the patch sets and each of the layers using features of the first positive example patch, the first negative example patch, and the second positive example patch of the patch set extracted from the feature map generated by the layer; Calculating the patch-wise loss by calculating a weighted sum of the losses calculated for each of the patch sets and for each of the layers; 26. The computer-readable medium of claim 25, wherein in the weighted sum, for a feature map whose size of a subregion of the input image corresponding to one cell of the feature map is closest to the size of the first positive example patch, a greater weight is set for a loss calculated using features extracted from that feature map than for other losses. [Explanation of symbols]

[0111] 10 Input images 12 subregion 20 Output images 22 Positive example patches 30 First feature map 40 Second feature map 50 training images 52 positive example patches 54 negative example patches 70 Generative Adversarial Networks 80 Identification Model 90 Label 100 Image Conversion Models 110 Feature Extraction Model 120 Image Generation Model 200 feature maps 210 Cells 300 Extraction Probability Map 310 Specific area 500 Computers 502 Bus 504 Processor 506 Memory 508 Storage Devices 510 Input / Output Interface 512 Network Interface 2000 Model Training Device 2020 Acquisition Department 2040 Training Execution Department

Claims

1. a training execution unit for training an image transformation model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment; An acquisition unit that acquires a training image that is a first image used for the training, the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, The training execution unit includes: inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Generate one or more patch sets, which are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region at a position in the output image corresponding to the first positive example patch; training the image transformation model using the first feature map, the second feature map, and a patch-wise loss calculated using the patch set; The training execution unit, in generating the patch set, extracts the first positive example patch from a specific region that represents a specific type of object among regions included in the training image; the training execution unit generates an extraction probability map representing an extraction probability, which is a probability that each of a plurality of partial regions of the training image will be extracted as the first positive example patch, and extracts the first positive example patch from the training image according to the extraction probability represented by the extraction probability map, thereby extracting the first positive example patch from the specific region with a focus on the first positive example patch; The training execution unit includes: Identifying a sub-class representing a finer classification than the specific class for the object represented by the specific region included in the training image; In the extraction probability map, the extraction probability for each partial region is set to be higher the higher the degree of overlap between the partial region and the specific region and the lower the occurrence frequency of the sub-type of the object represented by the specific region that overlaps with the partial region.

2. 2. The model training device according to claim 1, wherein the training execution unit calculates the patch-wise loss such that a magnitude of the patch-wise loss decreases as a similarity between a feature corresponding to the first positive example patch and a feature corresponding to the second positive example patch increases, and that a magnitude of the patch-wise loss increases as a similarity between a feature corresponding to the first negative example patch and a feature corresponding to the second positive example patch increases.

3. 1. A computer-implemented control method comprising: a training step of training the image transformation model to generate, from an input image representing a scene in a first environment, an output image representing said scene in a second environment; and acquiring a training image, which is a first image used for the training. the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Generate one or more patch sets, which are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region at a position in the output image corresponding to the first positive example patch; training the image transformation model using the first feature map, the second feature map, and a patch-wise loss calculated using the patch set; In the training execution step, in generating the patch set, the first positive example patch is extracted from a specific region representing a specific type of object among regions included in the training image; In the training execution step, an extraction probability map is generated for each of a plurality of partial regions of the training image, which indicates an extraction probability that the partial region is extracted as the first positive example patch, and the first positive example patch is extracted from the training image according to the extraction probability indicated by the extraction probability map, thereby extracting the first positive example patch from the specific region with a focus on the extraction; In the training execution step, Identifying a sub-class representing a finer classification than the specific class for the object represented by the specific region included in the training image; A control method in which, in the extraction probability map, for each partial region, the extraction probability is set to be higher the higher the degree of overlap between the partial region and the specific region and the lower the occurrence frequency of the sub-type of the object represented by the specific region that overlaps with the partial region.

4. On the computer, a training step of training the image transformation model to generate, from an input image representing a scene in a first environment, an output image representing said scene in a second environment; acquiring a training image, which is a first image used for the training; the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Generate one or more patch sets, which are a set of a first positive example patch and a first negative example patch that are partial regions of the training image, and a second positive example patch that is a partial region at a position in the output image corresponding to the first positive example patch; training the image transformation model using the first feature map, the second feature map, and a patch-wise loss calculated using the patch set; In the training execution step, in generating the patch set, the first positive example patch is extracted from a specific region representing a specific type of object among regions included in the training image; In the training execution step, an extraction probability map is generated for each of a plurality of partial regions of the training image, which indicates an extraction probability that the partial region is extracted as the first positive example patch, and the first positive example patch is extracted from the training image according to the extraction probability indicated by the extraction probability map, thereby extracting the first positive example patch from the specific region with a focus on the extraction; In the training execution step, Identifying a sub-class representing a finer classification than the specific class for the object represented by the specific region included in the training image; The program sets, in the extraction probability map, for each partial region, a higher extraction probability the higher the degree of overlap between the partial region and the specific region and the lower the occurrence frequency of the sub-type of the object represented by the specific region that overlaps with the partial region.

5. a training execution unit for training an image transformation model to generate an output image representing a scene in a second environment from an input image representing the scene in a first environment; An acquisition unit that acquires a training image that is a first image used for the training, the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, The training execution unit includes: inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Extracting a plurality of patch sets, each of which is a set of a first positive example patch and a first negative example patch that are a part of the training image, and a second positive example patch that is a region in the output image at a position corresponding to the first positive example patch; calculating a loss for each of the patch sets such that the magnitude of the loss decreases as the similarity between a feature amount corresponding to the first positive example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases, and the magnitude of the loss increases as the similarity between a feature amount corresponding to the first negative example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases; Using the loss calculated for each of the patch sets, calculate a patch-wise loss such that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific region representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific region; A model training device that uses the patch-wise loss to train the image transformation model.

6. 1. A computer-implemented control method comprising: a training step of training the image transformation model to generate, from an input image representing a scene in a first environment, an output image representing said scene in a second environment; and acquiring a training image, which is a first image used for the training. the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Extracting a plurality of patch sets, each of which is a set of a first positive example patch and a first negative example patch that are a part of the training image, and a second positive example patch that is a region in the output image at a position corresponding to the first positive example patch; calculating a loss for each of the patch sets such that the magnitude of the loss decreases as the similarity between a feature amount corresponding to the first positive example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases, and the magnitude of the loss increases as the similarity between a feature amount corresponding to the first negative example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases; Using the loss calculated for each of the patch sets, calculate a patch-wise loss such that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific region representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific region; A control method, training the image transformation model using the patch-wise loss.

7. On the computer, a training step of training the image transformation model to generate, from an input image representing a scene in a first environment, an output image representing said scene in a second environment; acquiring a training image, which is a first image used for the training; the image transformation model includes a feature extraction model that extracts features from each of a plurality of partial regions of the input image to generate a feature map, and an image generation model that generates the output image from the feature map generated by the feature extraction model, In the training execution step, inputting the training image into the image transformation model to obtain a first feature map which is a feature map of the training image and the output image; inputting the output image into the feature extraction model to obtain a second feature map that is a feature map of the output image; Extracting a plurality of patch sets, each of which is a set of a first positive example patch and a first negative example patch that are a part of the training image, and a second positive example patch that is a region in the output image at a position corresponding to the first positive example patch; calculating a loss for each of the patch sets such that the magnitude of the loss decreases as the similarity between a feature amount corresponding to the first positive example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases, and the magnitude of the loss increases as the similarity between a feature amount corresponding to the first negative example patch indicated by the first feature map and a feature amount corresponding to the second positive example patch indicated by the second feature map increases; Using the loss calculated for each of the patch sets, calculate a patch-wise loss such that an influence of the loss calculated for the patch set including the first positive example patch extracted from within a specific region representing a specific type of object is greater than an influence of the loss calculated for the patch set including the first positive example patch extracted from outside the specific region; A program that uses the patch-wise loss to train the image transformation model.

Citation Information

Patent Citations

  • Information prediction system, information prediction method and information prediction program

    JP2016095651A

  • Image processing method, image recording method, image processing device, and image file format

    WO2007029443A1

  • Information processing device and image processing method

    WO2014162690A1