Information processing apparatus, information processing method, and information processing program

Abstract

To more appropriately perform super resolution of images of different domains.SOLUTION: An information processing apparatus includes: an image quality conversion section that converts a high-resolution image into a low-resolution image; a deterioration processing section that generates a deteriorated image by further performing deterioration processing on the low-resolution image; a learning section that causes a super-resolution model to learn so as to restore the high-resolution image from the deteriorated image; and a generation section that generates a high-resolution image by super-resolving the low-resolution image in which a high resolution pair image is not present using the super-resolution model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Techniques for generating a super-resolution satellite image of an input satellite image using a super-resolution model have been disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in the above-mentioned conventional techniques. For example, in the above-mentioned conventional techniques, when generating a super-resolution satellite image of a satellite image, it is easy to perform super-resolution of an image in the same domain where there is a high-resolution image such as an aerial photograph. However, it may not work well for super-resolution of different images in a domain where there is no high-resolution image, and there is room for improvement in the implementation of super-resolution of different images in different domains. Therefore, means for more appropriately performing super-resolution of different images in different domains are required.

[0005] The present application has been made in view of the above, and an object thereof is to more appropriately perform super-resolution of different images in different domains.

Means for Solving the Problems

[0006] The information processing apparatus according to the present application includes an image quality conversion unit that converts a high-resolution image into a low-resolution image, a degradation processing unit that further performs degradation processing on the low-resolution image to generate a degraded image, a learning unit that causes a super-resolution model to learn to restore a high-resolution image from the degraded image, and a generation unit that uses the super-resolution model to super-resolve a low-resolution image for which no high-resolution paired image exists to generate a high-resolution image.

Effect of the Invention

[0007] According to one aspect of the embodiment, super-resolution of images with different domains can be more appropriately performed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0009] Hereinafter, a mode (hereinafter referred to as "embodiment") for implementing the information processing apparatus, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited by this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and duplicate explanations are omitted.

[0010] 〔1. Overview of the Information Processing System〕 First, referring to FIG. 1, the overview of the information processing system according to the embodiment will be described. FIG. 1 is an explanatory diagram showing the overview of the information processing system according to the embodiment. As shown in FIG. 1, the information processing system 1 according to the embodiment includes a terminal device 10 and a server device 100. These various devices are connected to each other via a network N, either wired or wirelessly, so that they can communicate with each other. As a result, the terminal device 10 can cooperate with the server device 100. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0011] The terminal device 10 is an information processing device used by a user U. For example, the terminal device 10 is a smart device such as a smartphone (smartphone) or a tablet terminal, a mobile phone such as a feature phone (galaxy phone), a PC (Personal Computer), a PDA (Personal Digital Assistant), a game machine or an AV device equipped with a communication function, an information home appliance or a digital home appliance, a car navigation system, a wearable device (Wearable Device) such as a smartwatch or a head-mounted display, or smart glasses. Further, the terminal device 10 may be a house or building, a vehicle, a home appliance product, an electronic device, etc. corresponding to the IoT (Internet of Things).

[0012] In this embodiment, the terminal device 10 is a smart device such as a smartphone or a tablet terminal used by the user U, and is a portable terminal device that can communicate with any server device via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5G mobile communication system). Further, the terminal device 10 has a screen such as a liquid crystal display and has a screen having a touch panel function, and receives various operations on display data such as content, such as a tap operation, a slide operation, and a scroll operation, from the user U using a finger, a stylus, or the like. Note that, among the operations performed on the screen, an operation performed on the area where the content is displayed may be regarded as an operation on the content. Further, the terminal device 10 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

[0013] Further, such a terminal device 10 can be connected to the network N via a wireless communication network such as LTE, 4G, or 5G, or a short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network), and communicate with the server device 100.

[0014] The server device 100 is, for example, a computer such as a PC or a blade server, or a mainframe or a workstation. Note that the server device 100 may be realized by cloud computing.

[0015] In this embodiment, the server device 100 is an information processing device that cooperates with the terminal device 10 of each user U and provides an API (Application Programming Interface) service or the like for various applications (hereinafter referred to as apps) and various data to the terminal device 10 of each user U, and is realized by a computer, a cloud system, or the like.

[0016] Further, the server device 100 may be an information processing device that provides some kind of web service online to each user U's terminal device 10. For example, as a web service, the server device 100 may provide services such as Internet connection, search service, SNS (Social Networking Service), e-commerce (EC: Electronic Commerce), electronic payment, online game, online banking, online trading, accommodation and ticket reservation, video and music distribution, news, map, route search, route guidance, route information, operation information, weather forecast, etc. In fact, the server device 100 may cooperate with various servers that provide the above web services and mediate the web service, or be responsible for the processing of the web service.

[0017] In addition, the server device 100 can acquire user information about the user U. For example, as user information, the server device 100 acquires information (attribute information) regarding attributes of the user U such as the gender, age, and residential area of the user U. Further, the server device 100 can acquire information regarding attributes such as the demographics (demographic attributes), psychographics (psychological attributes), geographics (geographical attributes), and behavioral (behavioral attributes) of the user U. Also, the server device 100 may acquire, as user information, the segment or persona (portrait of a person) to which the user U belongs in the field of marketing. Then, the server device 100 stores and manages information (attribute information) regarding the attributes of the user U together with the identification information (such as user ID) indicating the user U.

[0018] In addition, the server device 100 acquires various types of history information (log data) indicating the actions of the user U from the terminal device 10 of the user U or from various servers or the like based on the user ID or the like. For example, the server device 100 acquires a position history, which is a history of the position and time of the user U, from the terminal device 10. In addition, the server device 100 acquires a search history, which is a history of search queries input by the user U, from a search server (search engine). In addition, the server device 100 acquires a browsing history, which is a history of content browsed by the user U, from a content server. In addition, the server device 100 acquires a purchase history (settlement history), which is a history of product purchases and settlement processing of the user U, from an e-commerce server or a settlement processing server. In addition, the server device 100 may acquire a listing history and a sales history, which are histories of the user U's listings on the marketplace, from an e-commerce server or a settlement processing server. In addition, the server device 100 acquires a posting history, which is a history of posts of the user U, from a posting server or an SNS server that provides a word-of-mouth posting service. Note that each of the above-mentioned various servers or the like may be the server device 100 itself. That is, the server device 100 may function as each of the above-mentioned various servers or the like.

[0019] In addition, the number of each device included in the information processing system 1 shown in FIG. 1 is not limited to that shown. For example, in FIG. 1, for the sake of simplification of illustration, only one terminal device 10 is shown, but this is merely an example and is not limiting, and two or more may be provided.

[0020] [2. Satellite Image Super-Resolution] In the present embodiment, the server device 100 super-resolves (enhances the image quality) a low-resolution image for which no high-resolution paired image exists. Therefore, the server device 100 realizes this by separately preparing a high-resolution aerial photograph, converting (degrading) it like a satellite image of a map application or the like using a GAN (Generative Adversarial Networks), and training the model in a state where a pair of high-resolution and low-resolution satellite images is pseudo-exist. Note that the map application is merely an example. Actually, it may be a car navigation application, a map site, or an application or site similar thereto.

[0021] Not only GAN but also CUT (Contrastive Learning for Unpaired Image-to-Image Translation) can perform domain conversion. Therefore, in practice, domain conversion can be carried out using GAN or CUT. In this embodiment, as an example, domain conversion is carried out using GAN.

[0022] First, as shown in FIG. 1, the server device 100 causes a conversion model to learn to degrade (reduce image quality) a high-resolution image such as an aerial photograph into a low-resolution image such as a satellite image of a map application or the like through machine learning using GAN (step S1).

[0023] For example, the server device 100 constructs a generator that generates a low-resolution image (low-resolution satellite image) obtained by degrading a high-resolution image (high-resolution aerial photograph) to an image with an arbitrary lower resolution (for example, 1 / 4 resolution) through machine learning using CycleGAN, and a generator that restores (generates) a high-resolution image from the low-resolution image. CycleGAN can learn the relationship from two different image datasets and perform image conversion without preparing a large number of paired images.

[0024] At this time, in a state where there is no high-resolution aerial photograph as the original image, it is difficult to perform conversion in the direction of increasing the amount of information such as conversion from a low-resolution satellite image to a high-resolution aerial photograph. Therefore, learning is first performed using images in the domain where high-resolution aerial photographs exist.

[0025] For the satellite image (the whole of Japan) of the map application, although the area is narrowed, high-resolution aerial photographs of the entire specific area (Nerima Ward) are publicly available as open data. Therefore, the server device 100 uses this and constructs a conversion model between the satellite image and the aerial photograph through machine learning using CycleGAN.

[0026] Images in both the domain of satellite images and aerial photos include urban areas, green spaces, ponds, roads, crosswalks, solar panels, etc. With the GAN model, domain conversion between high-resolution images (aerial photos) and low-resolution images (satellite photos) has become somewhat possible. In particular, the conversion from aerial photos to satellite photos required for learning was of a quality that was quite faithful visually (in appearance).

[0027] As a result, for the server device 100, for a low-resolution image (low-resolution satellite image) with reduced image quality from a high-resolution image (high-resolution aerial photo), since the high-resolution image (high-resolution aerial photo) exists as the original image, it becomes possible to have the conversion model learn a pair of the high-resolution aerial photo and the low-resolution satellite image.

[0028] However, if the server device 100 directly learns a low-resolution image (primary degraded image) whose image quality has simply been reduced by GAN from a high-resolution aerial photo, the model may be able to tell whether the low-resolution image is from GAN (a degraded image), and it may not operate properly. For example, the super-resolution model that uses it later may be able to tell whether the low-resolution image was generated by the GAN model. Therefore, it is important to add another degradation to the low-resolution image (primary degraded image).

[0029] Subsequently, the server device 100 generates a degraded image (secondary degraded image) with additional degradation due to noise, blur (blur: defocusing, blurring), compression, etc. added to the low-resolution image (primary degraded image) with reduced image quality from the high-resolution image (high-resolution aerial photo) (step S2). At this time, the server device 100 performs the addition of noise, blur, and compression artifacts by normal image processing instead of GAN.

[0030] Subsequently, the server device 100 causes the super-resolution model to learn to restore (generate) a high-resolution image from the degraded image (secondary degraded image) by machine learning (step S3).

[0031] That is, the server device 100 generates a low-resolution image (primary degraded image) by performing image quality degradation (primary degradation) on a high-resolution image (high-resolution aerial photograph) using a GAN model, and then generates a degraded image (secondary degraded image) by performing degradation (secondary degradation) on the low-resolution image (primary degraded image) by adding noise, blur, compression artifacts, etc. After that, a pair of the high-resolution image (high-resolution aerial photograph) and the degraded image (secondary degraded image) is used to train a super-resolution model.

[0032] In actual verification, when directly using the generator of CycleGAN as the super-resolution model, the quality was poor and it was unusable. That is, the GAN model is not suitable for the super-resolution model. Therefore, in this embodiment, as described above, a super-resolution model different from the model (GAN model) for degrading images is prepared, and super-resolution is learned using the training data (pair of high-resolution image and degraded image) generated using the GAN model.

[0033] As a result, a model that generates a degraded image with the image quality of a high-resolution image reduced by GAN and restores (generates) a high-resolution image from the degraded image operates well. Also, not only models such as CNN (Convolutional Neural Network) commonly used in super-resolution, but any super-resolution model including a diffusion model can be used. Currently, it is preferable that the super-resolution model is a non-GAN model, but in the future, there is also a possibility that the performance of the GAN model will improve and the GAN model can be used as a super-resolution model.

[0034] Then, the server device 100 inputs a low-resolution satellite image without a high-resolution paired image into this super-resolution model, and as a result of super-resolving the satellite image, generates a high-resolution image equivalent to a high-resolution aerial photograph (step S4). Thereby, the satellite image of a map application or the like without a paired image can be enhanced in image quality.

[0035] In this way, the server device 100 prepares pseudo paired images by generating low-resolution images from high-resolution images, and performs model learning to generate high-resolution images from low-resolution images. However, depending on the domain of the training data, such as the imaging target (what is being photographed), there are differences in effectiveness. For example, when learning with data that has many parking lots in the frame, it is not possible to appropriately convert residential area data into high-resolution images. Since the accuracy does not improve even if only images of the same domain are learned, images of various domains are used for learning.

[0036] [2-1. Image resolution improvement technology using various models] The domain is not the type of imaging target that a person recognizes from the image captured in the photo, but the type of imaging target that the model recognizes. When the model recognizes the imaging target, it is presumed that the model recognizes and discriminates the imaging target based on color and shape, but uses a different logic from that of humans. More specifically, it is possible to generate an image that is recognized as "an apple is being photographed" when viewed by a person, but is recognized as "a car is being photographed" when recognized by the model. Such an image can realize an image that not only simply reduces the resolution, but also controls the color arrangement and distribution of each pixel, so that when viewed by a human, it is recognized that "an apple" is the imaging target (i.e., the domain), but when input into the model, it is recognized that "a car" is the imaging target. Also, by controlling the light irradiated on the sign, it is possible to cause the model to misrecognize the sign.

[0037] Therefore, instead of simply using images with reduced resolution for learning, an image intentionally generated as a "modified image" is prepared using a model so that the model can recognize a domain different from the original domain while suppressing the external change of the "reference image", and learning is performed using this "modified image" to improve the accuracy of the model.

[0038] First, as shown in FIG. 1, the server device 100 causes the domain change model to learn to generate a changed image whose domain of the input reference image is changed by machine learning using a GAN (step S11). In practice, since both GAN (CycleGAN) and CUT can perform conversion between domains, it is possible to implement not only with GAN but also with CUT.

[0039] Subsequently, the server device 100 generates a changed image whose domain of the input reference image is changed using the above domain change model (step S12).

[0040] Note that the reference image is an image in which the model can correctly identify the imaging target. Also, although the changed image does not differ much from the reference image in appearance, it is an image in which the model misrecognizes the imaging target by subtly changing the color distribution or the like. For example, the changed image is an image that appears to be a residential area but is recognized as a pond by the model. By changing the domain, it is possible to prepare an image in which the type of imaging target that the model will recognize is changed.

[0041] That is, the server device 100 generates a changed image from the reference image to be processed using the domain change model. Thereby, the server device 100 changes the domain recognized by the model.

[0042] Subsequently, the server device 100 generates a degraded image in which the visual information of the above changed image is degraded (step S13).

[0043] For example, the server device 100 generates a degraded image by further adding degradation due to noise, blur (blur: defocusing, blurring), JPEG image compression, etc. to the changed image. At this time, the server device 100 further adds noise, blur, or compression artifacts to the changed image.

[0044] Note that the server device 100 may generate a low-resolution image with a reduced image quality of the changed image using a conversion model. For example, the server device 100 may generate a low-resolution image by reducing the resolution of the changed image, and may further generate a deteriorated image with additional noise, blur, or compression artifacts added to this low-resolution image.

[0045] Subsequently, the server device 100 causes the super-resolution model to learn to restore (generate) the reference image from the deteriorated image by machine learning (step S14).

[0046] Subsequently, the server device 100 uses the super-resolution model to generate an image corresponding to the reference image from the input image (step S15).

[0047] [2-2. Supplementary Note] Currently, there are techniques for changing the image domain, such as GAN and CUT. Using this technique, a domain change model can be constructed to convert a horse image into a zebra image. Therefore, using CycleGAN, the model is trained to generate a low-resolution image such as a satellite photo of a map application from a high-resolution image. Then, additional noise, blur, or compression artifacts are further added to the generated image of the trained model to generate a deteriorated image. Examples of deterioration methods include JPEG image compression and noise addition. Finally, the model is trained with pairs of high-resolution images and deteriorated images. By doing so, it has been found that super-resolution of images with different domains works well.

[0048] The server device 100 prepares images of multiple domains and causes the model to learn to perform mutual conversion. At this time, prepare a large number of images in a relatively wide area. For example, from a dataset containing many domains such as parking lots, intersections, or urban areas, forests, lakes, etc., separate them by domain and collect them appropriately at random. It is preferable to collect images of as many domains as possible.

[0049] Note that the server device 100 may also be configured to train a model to convert from aerial photographs to satellite photographs using CycleGAN and to train a model to restore (generate) aerial photographs from satellite photographs. At this time, it may also be trained to return to the original state. In this way, learning in which the domain changes naturally is performed.

[0050] 〔2-3. Mechanism of CycleGAN Learning〕 Referring to FIG. 2, the mechanism of CycleGAN learning will be described. FIG. 2 is a diagram showing an overall configuration image of CycleGAN.

[0051] In CycleGAN, four components, two generators and two discriminators, are simultaneously trained based on the definitions in the table in (a) of FIG. 2. In the example of FIG. 2, input_A is an apple and input_B is an orange.

[0052] FIG. 2(b) shows a simplified structure of CycleGAN. The input image is sent to a discriminator for evaluation, translated into another domain, evaluated by another discriminator, and then translated back again.

[0053] Generator G AB is a generator from A to B, which is a model for converting an image in domain A (for example, an apple) into an image in domain B (for example, an orange).

[0054] Generator G BA is a generator from B to A, which is a model for converting an image in domain B (for example, an orange) into an image in domain A (for example, an apple).

[0055] Discriminator A is a discriminator that evaluates A, which is a model for determining whether the input image is in domain A. It has been trained to output "1" when an image in domain A is input in advance and "0" when any other image is input.

[0056] The discriminator B is a discriminator that evaluates B, a model that determines whether the input image is in domain B, and has been trained to output "1" when an image in domain B is input in advance and "0" when any other image is input.

[0057] By learning these by the learning method shown in (b) of FIG. 2, the generator G AB (a model that converts domain A to domain B) and the generator G BA (a model that converts domain B to domain A) can be accurately realized.

[0058] For example, as a result of evaluating the input_A as the original image (an image of an apple) by the discriminator A, an estimated value [0, 1] is obtained. Also, from this original image, the generator G AB generates a translated image (an image in which the apple is replaced with an orange), and as a result of evaluating this translated image by the discriminator B, an estimated value [0, 1] is obtained. Then, from this translated image, the generator G BA generates a reconstructed image (an image of an apple).

[0059] Also, as a result of evaluating the input_B as the original image (an image of an orange) by the discriminator B, an estimated value [0, 1] is obtained. Also, from this original image, the generator G BA generates a translated image (an image in which the orange is replaced with an apple), and as a result of evaluating this translated image by the discriminator A, an estimated value [0, 1] is obtained. Then, from this translated image, the generator G AB generates a reconstructed image (an image of an orange).

[0060] Furthermore, learn so that the "original image" and the "reconstructed image", which are images returned to the original domain after being converted to different domains, do not change as much as possible. As a result, the generator will convert the domain more faithfully, the discriminator will be able to identify more accurately whether the generated forgery is genuine or not, and the conversion quality of the generator will improve.

[0061] [3. Configuration Example of Terminal Device] Next, the configuration of the terminal device 10 will be described with reference to FIG. 3. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. 3, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0062] (Communication Unit 11) The communication unit 11 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the server device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0063] (Display Unit 12) The display unit 12 is a display device that displays various types of information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (Organic Electro-Luminescent Display). Further, the display unit 12 is a touch panel type display, but is not limited thereto.

[0064] (Input Unit 13) The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons or the like for inputting characters, numbers, and the like. Note that the input unit 13 may be an input / output port (I / O port), a USB (Universal Serial Bus) port, or the like. Further, when the display unit 12 is a touch panel type display, a part of the display unit 12 functions as the input unit 13. Also, the input unit 13 may be a microphone or the like that receives voice input from the user U. The microphone may be wireless.

[0065] (Positioning Unit 14) The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites, and based on the received signals, acquires position information (for example, latitude and longitude) indicating the current position of the terminal device 10, which is the own device. That is, the positioning unit 14 positions the position of the terminal device 10. Note that GPS is merely an example of GNSS (Global Navigation Satellite System).

[0066] In addition to GPS, the positioning unit 14 can also position the position by various methods. For example, the positioning unit 14 may use various communication functions of the terminal device 10 as auxiliary positioning means for position correction or the like to position the position as follows.

[0067] (Wi-Fi positioning) For example, the positioning unit 14 positions the position of the terminal device 10 by using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication or the like, and positions the distance from nearby base stations or access points to position the position of the terminal device 10.

[0068] (Beacon positioning) In addition, the positioning unit 14 may position the position by using the Bluetooth (registered trademark) function of the terminal device 10. For example, the positioning unit 14 positions the position of the terminal device 10 by connecting to a beacon transmitter connected by the Bluetooth (registered trademark) function.

[0069] (Geomagnetic positioning) In addition, the positioning unit 14 positions the position of the terminal device 10 based on the geomagnetic pattern of the structure measured in advance and the geomagnetic sensor provided in the terminal device 10.

[0070] (RFID positioning) Also, for example, when the terminal device 10 has a function equivalent to an RFID (Radio Frequency Identification) tag used at a station ticket gate, a store, etc., or has a function of reading an RFID tag, the used location is recorded together with the information on which settlement, etc. was performed by the terminal device 10. The positioning unit 14 may position the location of the terminal device 10 by acquiring such information. Also, the location may be positioned by an optical sensor, an infrared sensor, etc. provided in the terminal device 10.

[0071] The positioning unit 14 may, as necessary, position the location of the terminal device 10 using one or a combination of the above-described positioning means.

[0072] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. Note that the connection may be either a wired connection or a wireless connection. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices. In the example shown in FIG. 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, a pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

[0073] Note that the above-described sensors 21 to 28 are merely examples and are not limited. That is, the sensor unit 20 may be configured to include some of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28.

[0074] The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10 such as the moving direction, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10 such as the inclination in three-axis directions based on the angular velocity, etc. of the terminal device 10. The pressure sensor 23 detects, for example, the atmospheric pressure around the terminal device 10.

[0075] Since the terminal device 10 is equipped with the above-described acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc., it is possible to measure the position of the terminal device 10 using technologies such as pedestrian dead reckoning (PDR) that utilize these sensors 21 to 23. As a result, it becomes possible to obtain position information indoors, which is difficult to obtain with a positioning system such as GPS.

[0076] For example, with a pedometer that uses the acceleration sensor 21, the number of steps, walking speed, and distance walked can be calculated. Also, by using the gyro sensor 22, it is possible to know the direction of travel, the direction of the line of sight, and the inclination of the body of the user U. Further, from the atmospheric pressure detected by the barometric pressure sensor 23, it is also possible to know the altitude and the floor number where the terminal device 10 of the user U is located.

[0077] The temperature sensor 24 detects, for example, the temperature around the terminal device 10. The sound sensor 25 detects, for example, the sound around the terminal device 10. The light sensor 26 detects the illuminance around the terminal device 10. The magnetic sensor 27 detects, for example, the geomagnetism around the terminal device 10. The image sensor 28 captures an image of the surroundings of the terminal device 10.

[0078] The above-described barometric pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the environment and situation around the terminal device 10 by detecting barometric pressure, temperature, sound, illuminance, or capturing an image of the surroundings. Also, it becomes possible to improve the accuracy of the position information of the terminal device 10 from the environment and situation around the terminal device 10.

[0079] (Control unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, input / output ports, etc., and various circuits. Further, the control unit 30 may be configured by hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 has a transmission unit 31, a reception unit 32, and a processing unit 33.

[0080] (Transmission unit 31) The transmission unit 31 can transmit, via the communication unit 11, various information input by the user U using the input unit 13, various information detected by the sensors 21 to 28 mounted on or connected to the terminal device 10, the position information of the terminal device 10 measured by the positioning unit 14, and the like to the server device 100.

[0081] (Reception unit 32) The reception unit 32 can receive, via the communication unit 11, various information provided from the server device 100 and requests for various information from the server device 100.

[0082] (Processing unit 33) The processing unit 33 controls the entire terminal device 10 including the display unit 12 and the like. For example, the processing unit 33 can output and display various information transmitted by the transmission unit 31 and various information from the server device 100 received by the reception unit 32 to the display unit 12.

[0083] (Storage unit 40) The storage unit 40 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. Various programs, various data, etc. are stored in such a storage unit 40.

[0084] [4. Configuration Example of Server Device] Next, with reference to FIG. 4, the configuration of the server device 100 according to the embodiment will be described. FIG. 4 is a diagram showing a configuration example of the server device 100 according to the embodiment. As shown in FIG. 4, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0085] (Communication Unit 110) The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Also, the communication unit 110 is connected to the network N by wire or wirelessly.

[0086] (Storage Unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD, an SSD, or an optical disk. The storage unit 120 may store attribute information and history information (log data) of the user U together with identification information (such as a user ID) indicating the user U.

[0087] (Control Unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array) when various programs (corresponding to an example of an information processing program) stored in the internal storage device of the server device 100 are executed using a storage area such as a RAM as a work area. In the example shown in FIG. 4, the control unit 130 includes an acquisition unit 131, a domain change unit 132, an image quality conversion unit 133, a degradation processing unit 134, a learning unit 135, and a generation unit 136.

[0088] (Acquisition Unit 131) The acquisition unit 131 acquires the search query input by the user U. For example, when the user U inputs a search query to a search engine or the like to perform a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. That is, the acquisition unit 131 acquires the keyword input by the user U to the search window of the search engine, site, or application via the communication unit 110.

[0089] In addition, the acquisition unit 131 acquires user information regarding the user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as a user ID) indicating the user U, the location information of the user U, the attribute information of the user U, etc. from the terminal device 10 of the user U. Also, the acquisition unit 131 may acquire identification information indicating the user U, the attribute information of the user U, etc. at the time of user registration of the user U. Then, the acquisition unit 131 stores the user information in the storage unit 120.

[0090] In addition, the acquisition unit 131 acquires various history information (log data) indicating the actions of the user U via the communication unit 110. For example, the acquisition unit 131 acquires various history information indicating the actions of the user U from the terminal device 10 of the user U or from various servers or the like based on the user ID or the like. Then, the acquisition unit 131 stores the various history information in the storage unit 120.

[0091] In addition, the acquisition unit 131 acquires image data from the terminal device 10 of the user U, another server device 100, or an external storage device or storage medium via the communication unit 110. Also, the acquisition unit 131 acquires the image data stored in the storage unit 120. For example, the acquisition unit 131 acquires a high-resolution image such as an aerial photograph. Also, the acquisition unit 131 acquires the input reference image. Also, the acquisition unit 131 acquires a low-resolution image such as a satellite image of a map application with a different domain.

[0092] (Domain change unit 132) The domain change unit 132 generates a changed image with the domain of the input reference image changed, using a domain change model trained to change the domain of the input reference image.

[0093] (Image quality conversion unit 133) The image quality conversion unit 133 converts a high-resolution image into a low-resolution image. For example, the image quality conversion unit 133 uses a conversion model trained to convert a high-resolution image into a low-resolution image by machine learning using a GAN to convert the high-resolution image into a low-resolution image.

[0094] Here, the image quality conversion unit 133 converts a high-resolution aerial photograph into a low-resolution satellite image, creating a state where a pair of high-resolution and low-resolution satellite images exists pseudo.

[0095] At this time, the image quality conversion unit 133 converts the high-resolution image into a low-resolution image with an arbitrarily lower resolution. For example, the image quality conversion unit 133 converts the high-resolution image into a low-resolution image degraded to an image with 1 / 4 resolution.

[0096] Note that the image quality conversion unit 133 may convert the changed image generated by the domain change unit 132 into a low-resolution image.

[0097] (Degradation processing unit 134) The degradation processing unit 134 further performs degradation processing on the low-resolution image to generate a degraded image. For example, the degradation processing unit 134 further performs degradation processing on the low-resolution image to generate a degraded image so that the low-resolution image is not detected by the model as being derived from a GAN.

[0098] Here, the degradation processing unit 134 further performs degradation processing on the low-resolution satellite image to generate a degraded image.

[0099] At this time, as the degradation processing, the degradation processing unit 134 adds degradation due to at least one of noise, blur, and compression to the low-resolution image to generate a degraded image.

[0100] Note that the degradation processing unit 134 may perform degradation processing on the modified image generated by the domain change unit 132 to generate a degraded image. Alternatively, the degradation processing unit 134 may perform degradation processing on the low-resolution image converted from the modified image by the image quality conversion unit 133 to generate a degraded image.

[0101] (Learning unit 135) The learning unit 135 causes the super-resolution model to learn to restore a high-resolution image from the degraded image. The super-resolution model may be a general CNN-based model or any super-resolution model including a diffusion model. At present, it is preferable that the super-resolution model is a non-GAN model. However, in the future, it is possible that the performance of the GAN model will improve and the GAN model can be used as the super-resolution model.

[0102] Here, the learning unit 135 causes the super-resolution model to learn to restore a high-resolution aerial photograph from the degraded image.

[0103] Note that the learning unit 135 may cause the super-resolution model to learn to restore a reference image from the degraded image.

[0104] (Generation unit 136) The generation unit 136 uses the super-resolution model to super-resolve a low-resolution image for which no high-resolution paired image exists and generate a high-resolution image.

[0105] Here, the generation unit 136 uses the super-resolution model to super-resolve a low-resolution satellite image of a map application and generate a high-resolution image equivalent to a high-resolution aerial photograph. For example, the generation unit 136 generates a high-resolution image corresponding to a high-resolution aerial photograph for a low-resolution satellite image of a map application for which no high-resolution aerial photograph exists.

[0106] Note that the generation unit 136 may use the super-resolution model learned to restore a reference image from the degraded image to generate an image corresponding to the reference image from the input image.

[0107] [5. Processing procedure] Next, the processing procedure by the server device 100 according to the embodiment will be described with reference to FIG. 5. FIG. 5 is a flowchart showing the processing procedure according to the embodiment. The following described processing procedure is repeatedly executed by the control unit 130 of the server device 100.

[0108] For example, as shown in FIG. 5, the acquisition unit 131 of the server device 100 acquires a high-resolution image such as an aerial photograph (step S101).

[0109] Subsequently, the image quality conversion unit 133 of the server device 100 uses a conversion model learned to convert a high-resolution image such as an aerial photograph into a low-resolution image by machine learning using a GAN, and converts the high-resolution image into a low-resolution image (step S102).

[0110] Subsequently, the degradation processing unit 134 of the server device 100 further performs a degradation process on the low-resolution image to generate a degraded image so that the model does not detect that the low-resolution image is derived from a GAN (step S103).

[0111] Subsequently, the learning unit 135 of the server device 100 causes the super-resolution model to learn to restore the high-resolution image from the degraded image (step S104).

[0112] Subsequently, the acquisition unit 131 of the server device 100 checks whether it can acquire high-resolution images of other domains (step S105). For the substantive examination, the acquisition unit 131 may check whether it has acquired high-resolution images of other domains.

[0113] At this time, when the acquisition unit 131 of the server device 100 can acquire high-resolution images of other domains (step S105: Yes), it acquires high-resolution images of other domains (returns to step S101).

[0114] On the contrary, when the acquisition unit 131 of the server device 100 cannot acquire high-resolution images of other domains (step S105: No), the learning of the super-resolution model is terminated and the process proceeds to the next process (proceeds to step S106).

[0115] Subsequently, the generation unit 136 of the server device 100 uses the super-resolution model to super-resolve the low-resolution satellite image of the map application where there is no high-resolution aerial photograph and generates a high-resolution image equivalent to the high-resolution aerial photograph (step S106).

[0116] [[6. Modification Example]] The terminal device 10 and the server device 100 described above may be implemented in various different forms other than the above-described embodiment. Therefore, below, a modification example of the embodiment will be described.

[0117] In the above embodiment, part or all of the processing executed by the server device 100 may actually be executed by the terminal device 10 (or an application operating on the terminal). For example, the processing may be completed in a stand-alone manner (by the terminal device 10 alone). In this case, it is assumed that the terminal device 10 has the functions of the server device 100 in the above embodiment. Also, in the above embodiment, since the terminal device 10 is in cooperation with the server device 100, from the perspective of the user U, the processing of the server device 100 also seems to be executed by the terminal device 10. That is, from another perspective, it can be said that the terminal device 10 includes the server device 100.

[0118] Also, in the above embodiment, the server device 100 may input and learn high-resolution images in units of meshes (area meshes) of the map application. For example, the server device 100 may sequentially select aerial photographs (high-resolution images) corresponding to satellite photographs (low-resolution images) for each mesh of the map application and cause the model to learn for each mesh of the map application. Note that the mesh size is arbitrary. For example, it may be a 100m mesh or the like.

[0119] Also, in the above-described embodiment, the server device 100 may randomly or sequentially select domains from an image including many domains as a method for preparing images for each domain. For example, the server device 100 may use each image obtained by splitting an image including many domains into images for each domain as a modified image, and may select these modified images randomly or in a predetermined order.

[0120] Also, in the above-described embodiment, the server device 100 may repeat the processes from domain change to super-resolution model learning until there are no candidates for domains to be learned. For example, when there are candidates for domains to be learned, the server device 100 changes the domain. Conversely, when there are no candidates for domains to be learned, the server device 100 ends the learning of the super-resolution model.

[0121] [7. Effects] As described above, the information processing apparatus (the terminal device 10 and the server device 100) according to the present application includes an image quality conversion unit 133 that converts a high-resolution image into a low-resolution image, a degradation processing unit 134 that further performs degradation processing on the low-resolution image to generate a degraded image, a learning unit 135 that causes a super-resolution model to learn to restore a high-resolution image from the degraded image, and a generation unit 136 that uses the super-resolution model to super-resolve a low-resolution image for which there is no high-resolution paired image to generate a high-resolution image.

[0122] For example, the image quality conversion unit 133 uses a conversion model learned to convert a high-resolution image into a low-resolution image by machine learning using a GAN to convert the high-resolution image into a low-resolution image. The degradation processing unit 134 further performs degradation processing on the low-resolution image to generate a degraded image so that the model cannot detect that the low-resolution image is derived from a GAN. The learning unit 135 causes the super-resolution model to learn to restore a high-resolution image from the degraded image.

[0123] Further, the image quality conversion unit 133 converts a high-resolution aerial photograph into a low-resolution satellite image, creating a state where a pair of a high-resolution image and a low-resolution image of the satellite image exists pseudo. The degradation processing unit 134 further performs degradation processing on the low-resolution satellite image to generate a degraded image. The learning unit 135 trains the super-resolution model to restore the high-resolution aerial photograph from the degraded image. The generation unit 136 uses the super-resolution model to super-resolve the low-resolution satellite image of the map application and generate a high-resolution image equivalent to the high-resolution aerial photograph.

[0124] As the degradation processing, the degradation processing unit 134 adds degradation caused by at least one of noise, blur, and compression to the low-resolution image to generate a degraded image.

[0125] The image quality conversion unit 133 converts the high-resolution image into a low-resolution image reduced to any lower resolution.

[0126] Moreover, the information processing apparatus according to the present application further includes a domain change unit 132 that generates a changed image with the domain of the input reference image changed using a domain change model trained to change the domain of the input reference image. The degradation processing unit 134 performs degradation processing on the changed image to generate a degraded image. The learning unit 135 trains the super-resolution model to restore the reference image from the degraded image. The generation unit 136 uses the super-resolution model to generate an image corresponding to the reference image from the input image.

[0127] The domain change unit 132 generates a changed image with the domain of the input reference image changed using the domain change model. The image quality conversion unit 133 converts the changed image into a low-resolution image. The degradation processing unit 134 performs degradation processing on the low-resolution image to generate a degraded image. The learning unit 135 trains the super-resolution model to restore the reference image from the degraded image.

[0128] By any one or combination of the above-described processes, the information processing apparatus according to the present application can more appropriately perform super-resolution of images with different domains. As a result, the satellite images of a map application or the like can be enhanced in image quality.

[0129] 〔8. Hardware Configuration〕 Also, the terminal device 10 and the server device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in FIG. 6, for example. Hereinafter, the server device 100 will be described as an example. FIG. 6 is a diagram showing an example of a hardware configuration. The computer 1000 has a form in which an output device 1010, an input device 1020 are connected, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected by a bus 1090.

[0130] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, etc., and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0131] The primary storage device 1040 is a memory device that primarily stores data used by the arithmetic unit 1030 for various operations, such as a RAM (Random Access Memory). Also, the secondary storage device 1050 is a storage device in which data used by the arithmetic unit 1030 for various operations and various databases are registered, and is realized by a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), NAS (Network Attached Storage), a file server, or the like.

[0132] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, and a printer. For example, it is realized by a connector of a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). Also, the input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, buttons, and a scanner, and is realized by, for example, USB or the like.

[0133] Also, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0134] Further, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0135] Note that the input device 1020 may be a device that reads information from an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0136] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic unit 1030, and also sends the data generated by the arithmetic unit 1030 via the network N to other devices.

[0137] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0138] For example, when the computer 1000 functions as the server device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 by executing the program loaded onto the primary storage device 1040. Further, the arithmetic unit 1030 of the computer 1000 may load a program acquired from other devices via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. Also, the arithmetic unit 1030 of the computer 1000 may cooperate with other devices via the network I / F 1080 and call and use the functions, data, etc. of the program from other programs of other devices.

[0139] 〔9. Others〕 The embodiments of the present application have been described above, but the present invention is not limited by the contents of these embodiments. Further, the constituent elements described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the above-described constituent elements can be combined as appropriate. Moreover, various omissions, substitutions, or changes of the constituent elements can be made without departing from the gist of the above-described embodiments.

[0140] In addition, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. Additionally, regarding the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0141] Also, each component of each illustrated device is a functional concept and does not necessarily need to be physically configured as shown in the figure. That is, the specific form of the distribution and integration of each device is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage situations.

[0142] For example, the above-described server device 100 may be implemented by a plurality of server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform or the like through an API (Application Programming Interface) or network computing.

[0143] Also, the above-described embodiments and modification examples can be appropriately combined as long as the processing contents do not conflict.

[0144] Also, the "section (section, module, unit)" described above can be read as "means", "circuit", etc. For example, the acquisition section can be read as an acquisition means or an acquisition circuit.

Explanation of Signs

[0145] 1 Information processing system 10 Terminal device 100 Server device 110 Communication section 120 Storage section 130 Control section 131 Acquisition section 132 Domain change section 133 Image quality conversion section 134 Degradation processing section 135 Learning section 136 Generation section

Claims

1. An image quality conversion unit that converts a high-resolution image into a low-resolution image, A degradation processing unit that further performs degradation processing on the low-resolution image to generate a degraded image, A learning unit that trains a super-resolution model to restore a high-resolution image from the degraded image, A generation unit that uses the super-resolution model to super-resolve a low-resolution image for which no high-resolution paired image exists and generate a high-resolution image, An information processing apparatus characterized by comprising the above.

2. The image quality conversion unit converts a high-resolution image into a low-resolution image using a conversion model trained by machine learning using a GAN to convert a high-resolution image into a low-resolution image, The degradation processing unit further performs degradation processing on the low-resolution image to generate a degraded image so that the model cannot detect that the low-resolution image is derived from a GAN, The learning unit trains the super-resolution model to restore a high-resolution image from the degraded image The information processing apparatus according to claim 1, characterized by the above.

3. The image quality conversion unit converts a high-resolution aerial photograph into a low-resolution satellite image to create a state where a pair of high-resolution and low-resolution satellite images exists pseudo, The degradation processing unit further performs degradation processing on the low-resolution satellite image to generate a degraded image, The learning unit trains the super-resolution model to restore a high-resolution aerial photograph from the degraded image, The generation unit uses the super-resolution model to super-resolve a low-resolution satellite image of a map application and generate a high-resolution image equivalent to a high-resolution aerial photograph The information processing apparatus according to claim 1, characterized by the above.

4. The degradation processing unit, as the degradation processing, adds degradation due to at least one of noise, blur, and compression to the low-resolution image to generate a degraded image The information processing apparatus according to claim 1, characterized by the above.

5. The image quality conversion unit converts a high-resolution image into a low-resolution image reduced to an arbitrary lower resolution The information processing apparatus according to claim 1, characterized by the above.

6. A domain change unit that generates a changed image with the domain of the input reference image changed using a domain change model trained to change the domain of the input reference image, Further comprising, The degradation processing unit performs degradation processing on the changed image to generate a degraded image, The learning unit trains the super-resolution model to restore the reference image from the degraded image, The generation unit generates an image corresponding to the reference image from the input image using the super-resolution model. The information processing apparatus according to claim 1, characterized in that.

7. The domain change unit generates a changed image in which the domain of the input reference image is changed using the domain change model. The image quality conversion unit converts the changed image into a low-resolution image. The degradation processing unit performs degradation processing on the low-resolution image to generate a degraded image. The learning unit causes the super-resolution model to learn so as to restore the reference image from the degraded image. The information processing apparatus according to claim 6, characterized in that.

8. An information processing method executed by an information processing apparatus, A conversion step of converting a high-resolution image into a low-resolution image, A degradation processing step of further performing degradation processing on the low-resolution image to generate a degraded image, A learning step of causing the super-resolution model to learn so as to restore a high-resolution image from the degraded image, A generation step of using the super-resolution model to super-resolve a low-resolution image for which no high-resolution pair image exists to generate a high-resolution image. An information processing method, characterized by including.

9. A conversion procedure for converting a high-resolution image into a low-resolution image, A degradation processing procedure for further performing degradation processing on the low-resolution image to generate a degraded image, A learning procedure for causing the super-resolution model to learn so as to restore a high-resolution image from the degraded image, A generation procedure for using the super-resolution model to super-resolve a low-resolution image for which no high-resolution pair image exists to generate a high-resolution image. An information processing program, characterized by causing a computer to execute.

Images