Resolution conversion device, resolution conversion method, and computer program for resolution conversion

Abstract

To provide a resolution conversion device capable of properly increasing the resolution of an image to a higher level.SOLUTION: A resolution conversion device comprises: a selection unit 31 which manages mutually different photographing situations and selects a super-resolution model corresponding to the photographing situations when an image was generated from among a plurality of super-resolution models for improving resolution; and a super-resolution processing unit 32 which generates a high-resolution image with higher resolution than resolution of the generated image by inputting the image to the selected super-resolution model.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a resolution conversion device, a resolution conversion method, and a computer program for resolution conversion that convert the resolution of an image.

Background Art

[0002] Techniques for converting the resolution of an image have been proposed (see Patent Document 1).

[0003] The information processing apparatus disclosed in Patent Document 1 converts the resolution of an input image into a low-resolution image having a lower predetermined resolution, and classifies the category of the low-resolution image using a category classification model that has learned the category to which the image belongs. Further, this information processing apparatus generates low-resolution masking data of the low-resolution image from the masking model corresponding to the classification using a masking model that has learned low-resolution masking data having a resolution equivalent to a predetermined resolution for each category. Then, this information processing apparatus converts the generated low-resolution masking data into high-resolution masking data using a super-resolution model that has learned high-resolution masking data having a resolution higher than at least the predetermined resolution corresponding to the low-resolution masking data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When an image to be subjected to high-resolution processing is collected, in order to reduce the data amount of the image, the image may be compressed using an irreversible data compression method. In such a case, the resolution of the object represented in the image to be subjected to high-resolution processing decreases compared to the original image before data compression. Therefore, it is required to appropriately increase the resolution of the image so that the resolution of the object represented in the image is improved.

[0006] Therefore, an object of the present invention is to provide a resolution conversion device capable of appropriately increasing the resolution of an image.

Means for Solving the Problem

[0007] According to one embodiment, a resolution conversion device is provided. This resolution conversion device includes a selection unit that corresponds to different shooting situations and selects a super-resolution model corresponding to the shooting situation when the image is generated from a plurality of super-resolution models for improving the resolution, and a super-resolution processing unit that generates a high-resolution image having a resolution higher than that of the image by inputting the image into the selected super-resolution model.

[0008] In this resolution conversion device, it is preferable that the image includes shooting situation information representing the shooting situation when the image is generated. And the selection unit preferably selects a super-resolution model corresponding to the shooting situation represented by the shooting situation information from among the plurality of super-resolution models.

[0009] Also, in this resolution conversion device, it is preferable that the selection unit selects a super-resolution model corresponding to the shooting situation when the image is generated by inputting the image into a classifier for classifying into shooting situations corresponding to each of the plurality of super-resolution models.

[0010] According to another form, a resolution conversion method is provided. This resolution conversion method corresponds to different shooting situations, and selects a super-resolution model corresponding to the shooting situation when the image is generated from a plurality of super-resolution models for improving the resolution of the image, and inputs the image into the selected super-resolution model, thereby generating a high-resolution image having a resolution higher than that of the image.

[0011] According to still another form, a computer program for resolution conversion is provided. This computer program for resolution conversion corresponds to different shooting situations, and selects a super-resolution model corresponding to the shooting situation when the image is generated from a plurality of super-resolution models for improving the resolution of the image, and includes instructions for causing a computer to generate a high-resolution image having a resolution higher than that of the image by inputting the image into the selected super-resolution model.

Advantages of the Invention

[0012] The resolution conversion device according to the present disclosure has the effect of being able to appropriately increase the resolution of an image.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0014] Hereinafter, with reference to the drawings, a resolution conversion device, a resolution conversion method executed by the resolution conversion device, and a computer program for resolution conversion will be described. This resolution conversion device is configured to correspond to different shooting situations, and selects a super-resolution model corresponding to the shooting situation when the image to be resolution-converted is generated from a plurality of super-resolution models for improving the resolution of the image. Then, this resolution conversion device generates a high-resolution image having a resolution higher than that of the image by inputting the image to the selected super-resolution model.

[0015] Hereinafter, an example in which the resolution conversion device is applied to an image collection system that collects images generated by a camera mounted on a vehicle will be described. In this example, the resolution conversion device generates a high-resolution image having a resolution higher than that of the collected image in order to generate teacher data used for training an identifier for object detection or an identifier for semantic segmentation. As a result, for example, the details of an object that appears blurred in the collected image are pseudo-restored in the high-resolution image, and the object becomes clearly visible. However, the resolution conversion device according to the present disclosure is not limited to this example, and may be applied to various applications that require generating a high-resolution image having a resolution higher than that of the image to be subjected to the resolution conversion process.

[0016] FIG. 1 is a schematic configuration diagram of an image collection system in which a resolution conversion device is implemented. In the present embodiment, the image collection system 1 includes at least one vehicle 2 and a server 3 which is an example of a resolution conversion device. Each vehicle 2 is connected to the server 3 via the wireless base station 5 and the communication network 4 by accessing the wireless base station 5 connected via, for example, a communication network 4 to which the server 3 is connected and a gateway (not shown). Note that in FIG. 1, only one vehicle 2 is shown for simplicity, but the image collection system 1 may have a plurality of vehicles 2. Similarly, in FIG. 1, only one wireless base station 5 is shown, but a plurality of wireless base stations 5 may be connected to the communication network 4.

[0017] Figure 2 is a schematic configuration diagram of the vehicle 2. The vehicle 2 includes a camera 11, a GPS receiver 12, a wireless communication terminal 13, and a data acquisition device 14. The camera 11, the GPS receiver 12, the wireless communication terminal 13, and the data acquisition device 14 are communicably connected via an in-vehicle network compliant with a standard such as a controller area network.

[0018] The camera 11 is an example of an imaging unit for photographing the surroundings of the vehicle 2, and includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of an area to be photographed on the two-dimensional detector. The camera 11 is attached, for example, inside the passenger compartment of the vehicle 2 so as to face the front of the vehicle 2. Then, the camera 11 photographs the front area of the vehicle 2 at a predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second), and generates an image in which the front area is captured. The image obtained by the camera 11 may be a color image or a gray image. Note that a plurality of cameras 11 having different photographing directions or focal lengths may be provided in the vehicle 2.

[0019] Each time the camera 11 generates an image, it outputs the generated image to the data acquisition device 14 via the in-vehicle network.

[0020] The GPS receiver 12 receives GPS signals from GPS satellites at a predetermined cycle, and measures the self-position of the vehicle 2 based on the received GPS signals. Then, the GPS receiver 12 outputs positioning information representing the measurement result of the self-position of the vehicle 2 based on the GPS signals to the data acquisition device 14 via the in-vehicle network at a predetermined cycle. Note that the vehicle 2 may have a receiver compliant with a satellite positioning system other than the GPS receiver 12. In this case, the receiver may measure the self-position of the vehicle 2.

[0021] The wireless communication terminal 13 is an example of a communication unit and is a device that executes wireless communication processing compliant with a predetermined wireless communication standard. For example, by accessing the wireless base station 5, it is connected to the server 3 via the wireless base station 5 and the communication network 4. Then, the wireless communication terminal 13 generates an uplink wireless signal including image data received from the data acquisition device 14. Then, the wireless communication terminal 13 transmits the uplink wireless signal to the wireless base station 5 to transmit the image data to the server 3.

[0022] The data acquisition device 14 temporarily stores the image generated by the camera 11, and transmits an image that satisfies a predetermined condition among the stored images to the server 3 via the wireless communication terminal 13. Hereinafter, the image transmitted to the server 3 may be referred to as a transmission target image.

[0023] For example, the data acquisition device 14 sets a series of images generated by the camera 11 at the timing when it is notified from an electronic control unit (ECU, not shown) that controls the running of the vehicle 2 that the deceleration of the vehicle 2 has become equal to or greater than a predetermined threshold value and within a predetermined period before and after that as the transmission target image. Alternatively, the data acquisition device 14 may set an image generated by the camera 11 when the position of the vehicle 2 measured by the GPS receiver 12 is included in a predetermined collection target area as the transmission target image.

[0024] The data acquisition device 14 includes, as metadata, shooting situation information indicating the shooting situation when the transmission target image was generated in the transmission target image. The shooting situation information includes at least one of, for example, the generation date and time of the transmission target image, the position of the vehicle 2 when the transmission target image was generated, and weather information indicating the weather around the vehicle 2 when the transmission target image was generated. Note that the position of the vehicle 2 when the transmission target image was generated is measured by the GPS receiver 12. Also, the weather information is acquired, for example, from a weather server (not shown) via the wireless communication terminal 13.

[0025] Furthermore, the data acquisition device 14 may reduce the data amount of the transmission target image by compressing the transmission target image according to a predetermined image compression method. The predetermined image compression method may be either an irreversible compression method or a reversible compression method, and for example, it can be JPEG or EXIF.

[0026] At a predetermined timing, the data acquisition device 14 transmits the image data including the transmission target image and the related shooting situation information to the server 3 via the wireless communication terminal 13. The predetermined timing can be, for example, a preset time, or the timing when the ignition switch of the vehicle 2 is turned off. Alternatively, the predetermined timing may be the timing when the data amount of the transmission target image stored in the data acquisition device 14 and not yet transmitted to the server 3 reaches a predetermined upper limit value.

[0027] Next, the server 3, which is an example of the resolution conversion device, will be described. FIG. 3 is a hardware configuration diagram of the server 3, which is an example of the resolution conversion device. The server 3 includes a communication interface 21, a storage device 22, a memory 23, and a processor 24. The communication interface 21, the storage device 22, and the memory 23 are connected to the processor 24 via signal lines. The server 3 may further include an input device such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0028] The communication interface 21 is an example of the communication unit and has an interface circuit for connecting the server 3 to the communication network 4. And the communication interface 21 is configured to be communicable with the vehicle 2 via the communication network 4 and the wireless base station 5. That is, the communication interface 21 delivers the image data received from the vehicle 2 via the wireless base station 5 and the communication network 4 to the processor 24.

[0029] The storage device 22 is an example of a storage unit, and has, for example, a hard disk device or an optical recording medium and its access device. The storage device 22 stores various data and information used in the resolution conversion process. For example, the storage device 22 stores the image data received from each vehicle 2, and for each of a plurality of super-resolution models, a parameter set that defines the super-resolution model. Each super-resolution model is associated with a different shooting situation, and is used to improve the resolution of an image obtained under the corresponding shooting situation. Further, the storage device 22 stores information for classifying the shooting situation (such as the time indicating the boundary between the daytime and nighttime zones for each date, map information representing the geographical range for each country, etc.). Alternatively, the storage device 22 may store a parameter set that defines a classifier used for classifying an image to be subjected to resolution conversion. Furthermore, the storage device 22 may store a computer program for executing the resolution conversion process, which is executed on the processor 24. Furthermore, the storage device 22 may store the generated high-resolution image.

[0030] The memory 23 is another example of a storage unit, and has, for example, a non-volatile semiconductor memory and a volatile semiconductor memory. The memory 23 temporarily stores various data generated during the execution of the resolution conversion process.

[0031] The processor 24 is an example of a control unit, and has one or more CPUs (Central Processing Units) and its peripheral circuits. The processor 24 may further have other arithmetic circuits such as a logical arithmetic unit or a numerical arithmetic unit. The processor 24 executes the resolution conversion process.

[0032] FIG. 4 is a functional block diagram of the processor 24 related to the resolution conversion process. The processor 24 includes a selection unit 31 and a super-resolution processing unit 32. Each of these units included in the processor 24 is a functional module realized, for example, by a computer program operating on the processor 24. Alternatively, each of these units included in the processor 24 may be a dedicated arithmetic circuit provided in the processor 24. The processor 24 executes a resolution conversion process on the image included in the image data for each image data.

[0033] The selection unit 31 selects a super-resolution model corresponding to the shooting situation when the image to be subjected to the resolution conversion process is generated from among a plurality of super-resolution models.

[0034] Each of the plurality of super-resolution models is configured to output a high-resolution image with improved resolution for the input image. Each of the plurality of super-resolution models may be a model having the same architecture, or all or some of the plurality of super-resolution models may be models having different architectures from each other. For example, each of the plurality of super-resolution models can be assumed to be composed of a so-called deep neural network (DNN). More specifically, each of the plurality of super-resolution models can be a DNN having a convolutional neural network (CNN)-type architecture such as Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR) or SRResNet, for example. Alternatively, all or some of the plurality of super-resolution models may be DNNs that utilize attention such as RCAN. Or, all or some of the plurality of super-resolution models may be models according to a method other than DNN. These super-resolution models correspond to different shooting situations. Each of these super-resolution models is pre-trained using a plurality of teacher images obtained under the shooting situation corresponding to the super-resolution model according to a predetermined learning method such as the error backpropagation method. Therefore, these super-resolution models can generate and output a high-resolution image such that the resolution of the object represented in the image obtained under the corresponding shooting situation is improved.

[0035] In this embodiment, the shooting situation is classified according to temporal conditions. In this case, the shooting situation is classified, for example, into a daytime period and a nighttime period. As a result, it becomes possible to prepare each super-resolution model so as to appropriately increase the resolution of the appearance features of an object whose appearance changes between the daytime period and the nighttime period. Alternatively, the shooting situation may be classified according to geographical conditions at the time of image generation. In this case, the shooting situation is classified, for example, for each country such as Japan, the United States, and Germany. As a result, it becomes possible to prepare each super-resolution model so as to appropriately increase the resolution of an object represented by different characters for each country. Furthermore, the shooting situation may be classified according to the weather at the time of image generation. In this case, the shooting situation may be classified, for example, into when it is raining or snowing and other times. As a result, it becomes possible to prepare each super-resolution model so as to appropriately increase the resolution of the appearance features of an object whose appearance changes between when it is raining or snowing and other weather conditions. Furthermore, the shooting situation may be classified by a combination of at least two of temporal conditions, geographical conditions, and weather.

[0036] The selection unit 31 refers to the shooting situation information included in the image and selects a super-resolution model corresponding to the shooting situation represented by the shooting situation information from among a plurality of super-resolution models. By selecting the super-resolution model in this way, the selection unit 31 can appropriately select the super-resolution model corresponding to the shooting situation at the time of image generation.

[0037] For example, when the shooting situation information included in the image includes the generation date and time of the image, the selection unit 31 determines whether the generation date and time is included in the daytime period or the nighttime period. Then, when the generation date and time of the image is included in the daytime period, the selection unit 31 selects the super-resolution model corresponding to the daytime period, while when the generation date and time of the image is included in the nighttime period, the selection unit 31 selects the super-resolution model corresponding to the nighttime period.

[0038] Also, when the shooting situation information included in the image contains the position of the vehicle 2 when the image was generated, the selection unit 31 refers to the position of the vehicle 2 and the map information to identify the country where the vehicle 2 was located when the image was generated. Then, the selection unit 31 selects a super-resolution model corresponding to the identified country.

[0039] Alternatively, when the shooting situation information included in the image contains weather information, the selection unit 31 selects a super-resolution model corresponding to the weather indicated by the weather information.

[0040] According to a modification, the selection unit 31 may select a super-resolution model corresponding to the shooting situation when the image was generated by inputting the image to a classifier for classifying into shooting situations corresponding to each of a plurality of super-resolution models. The selection unit 31 can use a DNN having a CNN-type architecture as such a classifier. Alternatively, the classifier may be configured according to other methods used for image classification. Thereby, the shooting situation is more appropriately classified according to the information represented in the image. Therefore, the selection unit 31 can appropriately select a super-resolution model corresponding to the shooting situation at the time of image generation.

[0041] In this example, the classifier is configured to classify the image into one of a plurality of shooting situation classes by being pre-trained according to a predetermined learning method using a large number of teacher images obtained under various shooting situations. And for each class of shooting situations, a corresponding super-resolution model is prepared.

[0042] According to another modification example, for each of a plurality of shooting situations, a feature distribution representing the distribution of the feature vectors of the images may be stored in advance in the storage device 22. The feature vectors include, for example, a plurality of element values representing features of the image, such as the luminance of the image or the average value of each color, the variance of the luminance or each color, and the contrast. Further, the feature distribution can be, for example, a normal distribution having a dimension corresponding to the number of element values included in the feature vector. In this case, the selection unit 31 calculates the feature vector of the image and calculates the Mahalanobis distance between the feature vector and the feature distribution for each shooting situation. Then, the selection unit 31 selects the super-resolution model corresponding to the shooting situation in which the Mahalanobis distance is minimized.

[0043] The selection unit 31 notifies the super-resolution processing unit 32 of the selected super-resolution model.

[0044] The super-resolution processing unit 32 generates a high-resolution image having a resolution higher than the resolution of the input image by inputting the image to the super-resolution model selected by the selection unit 31. Thereby, for the image input to the super-resolution model, for example, a high-resolution image having several times higher resolution in each of the horizontal and vertical directions is generated.

[0045] FIG. 5 is a diagram for explaining the outline of the resolution conversion process according to the present embodiment. In this example, as the super-resolution models, a super-resolution model 501 corresponding to the daytime as the shooting situation and a super-resolution model 502 corresponding to the nighttime as the shooting situation are prepared. Here, for the image 510 to be super-resolved, the generation time of the image 510 is included as shooting situation information. And the generation time of the image 510 is included in the daytime. Therefore, among the super-resolution model 501 and the super-resolution model 502, the super-resolution model 501 for the daytime is selected. And by inputting the image 510 to the selected super-resolution model 501, a high-resolution image 520 is generated. In this way, by using the super-resolution model corresponding to the shooting situation when the image 510 is generated, an appropriate high-resolution image 520 is generated.

[0046] FIG. 6 is an operation flowchart of the resolution conversion process in server 3. The processor 24 of server 3 may execute the resolution conversion process for each target image according to the operation flowchart shown below.

[0047] The selection unit 31 of the processor 24 selects a super-resolution model corresponding to the shooting situation when the image to be subjected to the resolution conversion process is generated from among a plurality of super-resolution models (step S101). Then, the super-resolution processing unit 32 of the processor 24 generates a high-resolution image having a resolution higher than the resolution of the input image by inputting the image into the selected super-resolution model (step S102). Then, the processor 24 ends the resolution conversion process.

[0048] As described above, this resolution conversion device is configured to correspond to different shooting situations, and selects a super-resolution model corresponding to the shooting situation when the image to be subjected to resolution conversion is generated from among a plurality of super-resolution models for improving the resolution of the image. Then, this resolution conversion device generates a high-resolution image having a resolution higher than the resolution of the image by inputting the image into the selected super-resolution model. By generating a high-resolution image in this way, this resolution conversion device can appropriately increase the resolution of the image so as to pseudo-improve the resolution of the object represented in the image. Therefore, even if a part of the information represented in the image is missing due to the image being compressed in an irreversible manner, this resolution conversion device can pseudo-restore the information represented in the image.

[0049] A computer program for causing a computer to realize the functions of each part of the processor of the resolution conversion device according to the above embodiment or modification example may be provided in a form stored in a computer-readable recording medium. Note that the computer-readable recording medium can be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0050] As described above, those skilled in the art can make various modifications according to the implemented forms within the scope of the present invention.

Explanation of Signs

[0051] 1 Image acquisition system 2 Vehicle 11 Camera 12 GPS receiver 13 Wireless communication terminal 14 Data acquisition device 3 Server (resolution conversion device) 21 Communication interface 22 Storage device 23 Memory 24 Processor 31 Selection unit 32 Super-resolution processing unit 4 Communication network 5 Wireless base station

Claims

1. A selection unit that selects a super-resolution model corresponding to the shooting situation when an image is generated by a camera mounted on a vehicle from a plurality of super-resolution models that correspond to different shooting situations and improve resolution; A super-resolution processing unit that generates a high-resolution image having a resolution higher than that of the image by inputting the image into the selected super-resolution model; comprising; The shooting situation includes at least one of the time when the image is generated, the position of the vehicle when the image is generated, and the weather around the vehicle when the image is generated. A resolution conversion device.

2. The image includes shooting situation information representing the shooting situation when the image is generated. The selection unit selects a super-resolution model corresponding to the shooting situation represented by the shooting situation information from among the plurality of super-resolution models. The resolution conversion device according to claim 1.

3. The selection unit selects a super-resolution model corresponding to the shooting situation when the image is generated by inputting the image into a classifier for classifying the shooting situations corresponding to the plurality of super-resolution models. The resolution conversion device according to claim 1.

4. The shooting situation includes a first shooting situation and a second shooting situation. The architecture of the super-resolution model corresponding to the first shooting situation and the architecture of the super-resolution model corresponding to the second shooting situation are different from each other. The resolution conversion device according to any one of claims 1 to 3.

5. Select a super-resolution model corresponding to the shooting situation when the image is generated by a camera mounted on a vehicle from a plurality of super-resolution models that correspond to different shooting situations and improve the resolution of the image. Generate a high-resolution image having a resolution higher than that of the image by inputting the image into the selected super-resolution model. including that; The shooting situation includes at least one of the time when the image is generated, the position of the vehicle when the image is generated, and the weather around the vehicle when the image is generated. A resolution conversion method.

6. Select a super-resolution model corresponding to the shooting situation when the image is generated by a camera mounted on a vehicle from a plurality of super-resolution models that correspond to different shooting situations and improve the resolution of the image. By inputting the image into the selected super-resolution model, a high-resolution image having a resolution higher than that of the image is generated. Cause the computer to execute The shooting situation includes at least one of the time when the image was generated, the position of the vehicle when the image was generated, and the weather around the vehicle when the image was generated. A computer program for resolution conversion.

Images