Information processing device, control method thereof, and program

Abstract

The present invention provides a display technology for an image to be displayed that suppresses a decrease in user satisfaction when, for example, the aspect ratio of the image to be displayed and the display area are different. [Solution] The information processing device includes: a conversion unit that converts a first image to a second image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area; a position control unit that controls the position of the second image converted from the first image by the conversion unit in the display area; a generation unit that generates an extended image of the second image by generating a third image in the blank area of ​​the display area that does not overlap with the second image whose position is controlled by the position control unit; and an output unit that outputs the data of the extended image generated by the generation unit.

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, a control method thereof, and a program. [Background Art]

[0002] In cases such as when an image captured by a camera is displayed on a display of another device, or when rotating the vertical / horizontal orientation of a smartphone while displaying an image recorded on the smartphone, the aspect ratio of the image to be displayed may not match the aspect ratio of the display area. In such cases, if the resolution of the target display image is changed to match the vertical or horizontal length of the display area while maintaining the aspect ratio of the target display image, a blank area where no target display image exists is generated in the display area. Techniques for displaying a black belt in such a blank area are known. In addition, techniques such as blurring and copying the edge of the target display image and displaying the copied image in the blank area, or displaying other content in the blank area, are also generally known. Further, Patent Document 1 describes a technique for trimming the target display image to match the aspect ratio of the display area and displaying the trimmed image over the entire display area in order to prevent the generation of a blank area. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 9-55899 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, in the above-described conventional techniques, it is possible that the image displayed in the blank area is not recognized as having continuity or consistency with the target display image. Accordingly, a sense of discomfort is caused to the user, and as a result, user satisfaction may be reduced. In addition, with the technique described in Patent Document 1, the entire target display image is not displayed. Such a situation can also be a factor that reduces user satisfaction.

[0005] The present invention has been made in view of at least one of the above-mentioned problems, and provides a display technology for a target image that suppresses a decrease in user satisfaction when the aspect ratio of the target image and the display area are different. [Means for solving the problem]

[0006] According to one aspect of the present invention, A conversion unit that converts the first image into a second image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, In the display area, a position control unit controls the position of the second image converted from the first image by the conversion unit, A generation unit generates an extended image of the second image by generating a third image in the margin area of ​​the display area that does not overlap with the second image whose position is controlled by the position control unit, The system comprises an output unit that outputs the data of the extended image generated by the generation unit, It is an information processing device. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a display technology for a target image that suppresses a decrease in user satisfaction when the aspect ratio of the target image and the display area are different. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram of a display device according to one embodiment. [Figure 2] Functional block diagram of the image generation unit according to one embodiment. [Figure 3] Flowchart of the process according to one embodiment [Figure 4] Figure showing a display screen according to one embodiment. [Figure 5] Flowchart of the process according to one embodiment [Figure 6A]FIG. 1 is a diagram illustrating a display screen according to an embodiment [Figure 6B] FIG. 1 is a diagram illustrating a display screen according to an embodiment [Figure 7] FIG. 1 is a functional block diagram of an image generation unit according to an embodiment [Figure 8] FIG. 1 is a flowchart of processing according to an embodiment [Figure 9] FIG. 1 is a diagram illustrating a display screen according to an embodiment [Figure 10] FIG. 1 is a flowchart of processing according to an embodiment [Figure 11] FIG. 1 is a diagram illustrating a display screen according to an embodiment [Figure 12] FIG. 1 is a diagram illustrating a display screen according to an embodiment [Figure 13] FIG. 1 is a flowchart of processing according to an embodiment [Figure 14] FIG. 1 is a flowchart of processing according to an embodiment [Figure 15] FIG. 1 is a flowchart of processing according to an embodiment [Figure 16] FIG. 1 is a functional block diagram of an image generation unit according to an embodiment [Figure 17] FIG. 1 is a flowchart of processing according to an embodiment [Figure 18] FIG. 1 is a diagram illustrating a display screen according to an embodiment MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of the plurality of features are essential to the invention, and a plurality of features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0010] (First Embodiment) <Example of Hardware Configuration of Display Device> The hardware configuration of a display device 100 (an example of an "information processing device") will be described with reference to FIG. 1(B). The display device 100 is configured to include a CPU (Central Processing Unit) 112, a ROM 113, a RAM 114, and a recording medium 115. The display device 100 is also configured to include a GPU (Graphics Processing Unit) 116, an input device 117, a display device 118, and a communication interface 119. Each component is communicatively connected to each other via a system bus 111.

[0011] The CPU 112 and the GPU 116 are each one or more processors capable of executing programs. The GPU 116 is configured to execute specific calculations at higher speed than the CPU 112, and in recent years, it is particularly used to execute inference processing using neural networks at high speed. As an alternative to the GPU 116, an NPU (Neural Processing Unit) specialized for executing learning and inference processing using neural networks may be provided. Instead of using separate CPU 112 and GPU 116, an SoC (System on Chip) integrating a CPU and a GPU (and an NPU in some cases) may be provided.

[0012] Further, the CPU 112 implements each functional block described later with reference to FIG. 1(A) and FIG. 2 by, for example, loading a program stored in the ROM 113 into the RAM 114 and executing the program. The CPU 112 implements high-speed processing by using the GPU 116 for processing using neural networks. Note that the GPU 116 may have a dedicated RAM separate from the RAM 114.

[0013] ROM 113 is, for example, an electrically rewritable non-volatile memory. ROM 113 stores programs that the CPU 112 can execute, setting values, GUI data, and parameters for implementing a trained neural network (training model). RAM 114 is used to load programs executed by the CPU 112 and to store values ​​necessary during program execution. Furthermore, RAM 114 includes VRAM (Video RAM) for display control. The recording medium 115 is, for example, a semiconductor memory card or an SSD (Solid State Drive), and is also used as a recording location for input image data of the display device 100.

[0014] The input device 117 includes multiple operating parts, such as buttons, touch panels, or switches, that receive operational input to the display device 100. The input device 117 may also include one or more devices (such as sensors) for acquiring additional information during image generation. Specifically, the input device 117 may include, for example, a GPS receiver for acquiring the location information of the display device 100, or a clock for acquiring the generation date and time. However, the input device 117 is not limited to these devices.

[0015] The display device 118 is, for example, a liquid crystal display provided on the surface of the housing of the display device 100. The display device 118 may also be a touch display. The display device 118 displays, for example, image data drawn in the VRAM within the RAM 114, a menu screen, and information about the display device 100 (e.g., setting values, battery level, etc.).

[0016] The communication interface 119 is a circuit for communicating with an external device in accordance with one or more communication standards, and includes a connector for wired communication, an antenna for wireless communication, and a transmitting and receiving circuit. The display device 100 can transmit image data to an external device and receive data from an external device through this communication interface 119. Typical communication standards to which the communication interface 119 conforms include, but are not limited to, HDMI (registered trademark), USB, Bluetooth (registered publication), and wireless LAN (Wi-Fi).

[0017] <Example of display device function configuration> Next, the functional blocks of the display device 100 will be described using Figure 1(A). Throughout the drawings, functional blocks can be implemented by software, or a combination of software and hardware, except for parts that are clearly only achievable in hardware (e.g., recording media). For example, a functional block may be implemented by dedicated hardware such as an ASIC. Alternatively, a functional block may be implemented by one or more program-executable processors, such as a CPU, executing a program stored in memory. Multiple functional blocks may be implemented by a common configuration (e.g., one ASIC). Furthermore, hardware that implements some functions of one functional block may be included in hardware that implements other functional blocks.

[0018] As shown in Figure 1(A), the display device 100 is configured to include an image input unit 101, an image generation unit 102, and a display unit 103. The functions of the display device 100 are not limited to those realized by the function blocks shown in Figure 1(A). The image input unit 101 acquires image data recorded on the recording medium of the display device 100 as input image data (an example of a "first image"). Here, the image data may be, for example, an image taken with a digital camera or a smartphone.

[0019] Furthermore, the image input unit 101 outputs the resolution of the input image data to the image generation unit 102. In addition, the image input unit 101 may output to the image generation unit 102 some of the supplementary information recorded in the data file in which the input image data is stored (for example, one or more tags related to the shooting conditions and shooting circumstances of the captured image).

[0020] In this embodiment, the case in which the image input unit 101 acquires image data from a recording medium of the display device 100 is described as an example, but the invention is not limited to this example. For example, the image input unit 101 may acquire image data recorded on an external server or cloud that the display device 100 can communicate with. Alternatively, the image input unit 101 may acquire image data recorded on an external storage device such as a USB memory or external HDD connected to the display device 100.

[0021] The image generation unit 102 generates display image data using the input image data acquired from the image input unit 101. The image generation unit 102 then outputs the generated display image data to VRAM. The display unit 103 reads the display image data generated by the image generation unit 102 from VRAM and displays it on the display device.

[0022] Using Figure 2, a block illustrating an example of the functional configuration of the image generation unit 102 will be explained. The image generation unit 102 is composed of a resolution conversion unit 201, an image expansion unit 202, and an image output unit 203. The resolution conversion unit 201 converts the horizontal or vertical dimensions of the input image to the horizontal or vertical dimensions of the display area while maintaining the aspect ratio of the input image's resolution. The resolution conversion unit 201 then outputs the converted input image data to the image expansion unit 202.

[0023] The image extension unit 202 extends the original image by generating a new image that is continuous or consistent with the image data output from the resolution conversion unit 201, for example, using an image generation AI model. More specifically, the image generation AI model is a neural network generated using, for example, an image cropped from an existing image as input image data, and image data surrounding the cropped image from the existing image as training data. The parameters of this image generation AI model are, for example, image patterns or features. The image generation AI model trained in this way can infer patterns or features of surrounding images from the input image and generate a new image around the input image. For the learning method and inference process, well-known techniques related to image generation AI may be applied. The image generation AI model may be stored in advance in the display device 100, for example, or stored in an external device that the display device 100 can communicate with. Furthermore, the method for generating images around the input image is not limited to the above method, and may be a method that generates them according to a predetermined rule, for example. The image output unit 203 outputs the image data output from the resolution conversion unit 201 or the image extension unit 202 to VRAM. The input image, which has been augmented by the newly generated image in this way, will henceforth be referred to as the augmented image.

[0024] The functional blocks shown in Figures 1(A) and 2 are implemented by one or more of the hardware components shown in Figure 1(B). For example, the image generation unit 102 is mainly implemented by the CPU 112 and GPU 116. The RAM 114 is used as a temporary storage location for data to be processed, data being processed, and processing result data, while the ROM 113 is used as a reference point for pre-stored settings and programs, and is used for various processes.

[0025] <Operation during display processing> The display operation of the display device 100 will be explained using Figures 3 and 4(A) to 4(D). Note that the operation of each step in the flowchart in Figure 3 is achieved by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as needed.

[0026] Furthermore, the display operation of the display device 100 is performed in response to instructions received through the input device 117. For example, a list of thumbnails of input images may be displayed on the display device 118, and the system may respond to selection operations via the input device 117. Alternatively, the display operation of the display device 100 may be performed according to predetermined conditions other than instructions. For example, the display operation may be performed sequentially and automatically for multiple image data received from an external device via the communication interface 119.

[0027] In S301, the image input unit 101 acquires input image data. The image input unit 101 then outputs the acquired input image data and the associated information described above to the image generation unit 102. The image input unit 101 may also process the input image data to make it suitable for use by the image generation unit 102 before outputting it. In S302, the resolution conversion unit 201 of the image generation unit 102 acquires the resolution of the input image and the display area, respectively. The resolution conversion unit 201 then converts the resolution of the input image data so that the horizontal or vertical width of the input image matches the horizontal or vertical width of the display area, while maintaining the aspect ratio of the input image. The image after the resolution conversion in this way is an example of a "second image".

[0028] Figure 4(A) illustrates an input image acquired by the image input unit 101. The aspect ratio of the input image is 16:9. Figure 4(B) illustrates the display area of ​​the display device 118. The aspect ratio of the display area is 3:2. Figure 4(C) illustrates an image in which the input image from Figure 4(A) is placed on the display area of ​​Figure 4(B). At this time, the resolution conversion unit 201 performs a conversion that matches the horizontal width of the input image to the horizontal width of the display area while maintaining the aspect ratio of the input image. In this way, when the aspect ratios of the input image and the display area are different, the resolution conversion unit 201 executes S302, which creates areas around the input image where no image exists. In Figure 4(C), such areas are exemplified by regions (401, 402) above and below the display area. In the first embodiment, the subsequent processing S303 to S304 generates a new image for regions (401, 402). The input image is expanded in this way.

[0029] Let's return to the explanation of the flowchart in Figure 3. In S303, the resolution conversion unit 201 determines whether the aspect ratios of the input image acquired by the image input unit 101 and the display area are different. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S304. Otherwise, the resolution conversion unit 201 uses the input image as the display image, and the process proceeds to S305.

[0030] In S304, the image extension unit 202 (an example of a "position control unit") places the input image in the center of the display area, as shown in Figure 4(C), for example. This creates areas (401, 402) where the input image does not overlap. The image extension unit 202 then generates a new image (an example of a "third image") that is continuous or consistent with the input image for these areas (401, 402). The image extension unit 202 then combines the input image data and the newly generated image data to generate data for the display image (extended image). This makes it possible to display the image across the entire display area even when the aspect ratio of the input image and the resolution of the display area are different. Figure 4(D) shows an example of a display image displayed across the entire display area. Note that areas (401, 402) are an example of a "margin area".

[0031] In S305, the image output unit 203 stores the display image data output from the resolution conversion unit 201 in S302 or the image expansion unit 202 in S304 in the VRAM of RAM 114. In S306, the display unit 103 reads the display image data stored in the VRAM of RAM 114 and displays it on the display device 118. Then the process ends.

[0032] <One aspect of action / effect> As described above, the display device 100 generates a new image that is continuous or consistent with the input image when the aspect ratio of the input image and the display area are different. Then, the input image and the newly generated image are combined to create a display image. In this way, it becomes possible to display the image across the entire display area regardless of the difference in aspect ratio between the input image and the display area. Therefore, a decrease in user satisfaction is suppressed.

[0033] (Second Embodiment) Next, the display device 100 according to the second embodiment will be described. The display device 100 according to the second embodiment allows for arbitrary selection of the display method when the aspect ratio of the input image and the display area are different. In addition, the operation of the display device 100 when the aspect ratio of the display area is changed during image display will also be described in the second embodiment. Note that in the second embodiment, the processing of the image generation unit 102 differs from that of the first embodiment, but other processing is the same as in the first embodiment. Therefore, the same reference numerals are used for components and processes that are the same as or substantially the same as those described above in the first embodiment, and their descriptions are omitted.

[0034] <Operation during display processing> The display operation of the display device 100 according to the second embodiment will be explained using Figures 5, 6A, and 6B. First, an example of a display image will be explained using Figure 6A. Figure 6A(A) illustrates an input image acquired by the image input unit 101. The aspect ratio of the input image is 16:9. Figure 6A(B) illustrates the display area of ​​the display device 118. The aspect ratio of the display area is 3:2. Figure 6A(C) illustrates a display image in which the horizontal width of the input image in Figure 6A(A) is matched to the horizontal width of the display area in Figure 6A(B). The display image in Figure 6A(C) illustrates a state in which there are areas where no image exists in the upper and lower areas (601, 602) of the display area, and these areas (601, 602) are filled with black.

[0035] Furthermore, Figure 6A(D) illustrates a display image in which the vertical width of the input image exemplified in Figure 6A(A) is matched to the vertical width of the display area in Figure 6A(B). In other words, the display image in Figure 6A(D) is a display image in which the aspect ratio of the input image is matched to the aspect ratio of the display area by cropping the left and right regions of the input image. Thus, when the aspect ratio of the input image and the display area are different, methods for displaying the input image include blacking out the areas where the input image does not exist, or cropping a part of the input image and matching the aspect ratio.

[0036] On the other hand, Figure 6B illustrates the display images generated by each embodiment. Specifically, Figure 6B(E) is a display image (Figure 4(D)) generated by the image generation process according to the first embodiment. In the second embodiment, in addition to the display methods for the display images (Figures 6A(C) and 6A(D)), the user of the display device 100 can arbitrarily select a display method such as the display image (Figure B(E)) generated in the first embodiment.

[0037] Next, Figure 5 will be used to explain the flowchart of the display operation of the display device 100. Note that the operation of each step in the flowchart in Figure 5 is achieved by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as needed.

[0038] Furthermore, the display operation of the display device 100 is performed in response to instructions received through the input device 117. For example, a list of thumbnails of input image data may be displayed on the display device 118, and the system may respond to selection operations via the input device 117. Alternatively, the display operation of the display device 100 may be performed according to predetermined conditions other than instructions. For example, the display operation may be performed sequentially and automatically for multiple image data received from an external device via the communication interface 119.

[0039] Since steps S301 and S302 are the same as in the first embodiment, their explanation is omitted. The process then proceeds to S501. In S501, the resolution conversion unit 201 determines whether the aspect ratio of the resolution of the input image acquired by the image input unit 101 and the aspect ratio of the resolution of the display area are different. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S502; otherwise, the process proceeds to S305.

[0040] In S502, the resolution conversion unit 201 (an example of the "first determination unit") determines whether or not to generate an extended image by the image extension unit 202 using an arbitrary setting value specified by the user operating the display device 100. If the resolution conversion unit 201 determines that the user has permitted the generation of an extended image, the process proceeds to S304 and S305 in that order; otherwise, the process proceeds to S503.

[0041] In S503, the image output unit 203 determines whether to black out the surrounding area (601, 602) of the input image shown in Figure 6A(C), or to crop the input image to match the aspect ratio of the display area and display it in full. This determination is made using arbitrary settings specified by the user operating the display device 100. If the image output unit 203 determines to black out the area, the process proceeds to S504; if it determines to crop and display the image in full, the process proceeds to S505.

[0042] Furthermore, the method by which the user specifies the setting values ​​used for the determinations in S502 and S503 is as follows: If, after the start of the display process, S501 determines that the aspect ratio of the input image and the display area are different, the following process is executed. That is, the display unit 103 displays an arbitrary notification on the screen. Then, the CPU 112 accepts that the user will perform an operation on the content of that notification via an arbitrary operating component.

[0043] More specifically, the display unit 103 displays messages on the touch panel display indicating options for how to display the image, such as "display the image enlarged," "display the image with a black border," and "display the image cropped." The user specifies how to display the image by selecting one of these options via the touch panel. The CPU 112 then accepts this selection.

[0044] Alternatively, the display method can be arbitrarily specified in advance by adding a setting for the display method to the settings of the display device 100. Note that the user's specification of the image display method is not limited to these methods and may be specified by other methods. Furthermore, in the second embodiment, three display methods are arbitrarily selectable, but other display methods may also be arbitrarily selectable. Other display methods include, for example, displaying other information in the area (601, 602) surrounding the input image, or copying the edges of the input image instead of blacking them out and filling the area (601, 602) with the copied image.

[0045] In S504, the image output unit 203 adds a black image to the areas (601, 602) where no input image exists. Figure 6A(C) shows an example of a display image with black images added to areas (601, 602). The process then proceeds to S305.

[0046] In S505, the image output unit 203 crops the input image to match the aspect ratio of the display area's resolution. The cropping position of the input image is determined by aligning the center of the input image with the center of the display area, and ensuring that either the horizontal or vertical width of the input image is the same as the horizontal or vertical width of the display area. Alternatively, the cropping position may be arbitrarily determined by the user. In this way, a display method is achieved in which a portion of the input image is cropped to match the aspect ratio of the display area, as shown in Figure 6A(D). The process then proceeds to S305.

[0047] Steps S304 to S306 are the same as in the first embodiment, so their explanation is omitted. After the execution of S306, the process proceeds to S506. In S506, the display unit 103 determines whether the display state of the display image has been canceled, that is, whether the display process has ended. If the display unit 103 determines that the display process has ended, the process ends; otherwise, the process proceeds to S507. The conditions under which the display state of the display image is canceled include, for example, when the display device 118 is turned off, or when the display state of the display device 118 transitions to another mode. Another mode may be, for example, a mode in which a list of thumbnail images is displayed, or a mode in which a settings screen for configuring the display device 100 is displayed.

[0048] In S507, the display unit 103 (an example of the "second determination unit") determines whether the aspect ratio of the display area has changed while the display image is being displayed. If the display unit 103 determines that the aspect ratio has changed, the process returns to S302; otherwise, the process returns to S506.

[0049] Here, we will explain the details of S507. Figure 6B(F) illustrates the display area of ​​the display device 118 when the display device 100 is rotated by 90 degrees. The display device 100 according to the second embodiment has a function to automatically switch the screen display between portrait and landscape orientation depending on the orientation of the display device 118. Therefore, when the orientation of the display device 118 is rotated by 90 degrees, the aspect ratio of the display area is reversed. For example, the aspect ratio of the width and height in Figure 6A(B) is 3:2, while the aspect ratio of Figure 6B(F) after 90-degree rotation is 2:3. However, when the aspect ratio is changed in this way, areas (603, 604) where there is no input image occur at the top and bottom of the display area, as shown in Figure 6B(G).

[0050] On the other hand, Figure 6B(H) shows an example where the input image in Figure 6A(A) is displayed across the entire display area of ​​Figure 6A(F) even when the aspect ratio is changed by executing the flowchart shown in Figure 5. More specifically, if the aspect ratio of the display area is changed while the display image is being displayed, the change is detected by the determination process in S507. Then, S302 to S304 are executed again for the input image in Figure 6A(A), generating an image with a changed resolution (an example of a "fourth image"), and further generating an extended image (an example of a "second extended image") for this image. Therefore, even if the aspect ratio of the display area is changed, it is possible to display a display image that conforms to the changed aspect ratio.

[0051] The image output unit 203 may also highlight the new image generated by the image expansion unit 202. Highlighting may, for example, enclose the generated image in a frame. Furthermore, highlighting may be applied when the ratio of the area of ​​the newly generated image to the area of ​​the input image exceeds a predetermined threshold. Alternatively, highlighting may be applied when there is a predetermined or greater discrepancy between the aspect ratio of the input image and the aspect ratio of the newly generated image. Alternatively, highlighting may be configured to be arbitrarily specified by the user.

[0052] Furthermore, during the execution of the display processing according to the second embodiment, the display image data output by S305 may be stored in RAM 114 (an example of the "first storage unit"). Under the same conditions, steps S304, S504, and S505 may be omitted, and the display image data stored in RAM 114 may be used. For example, if the user requests the display of an extended image, the image extension unit 202 generates the extended image data shown in Figure 6B(E) in S304. Subsequently, if the display device 118 of the display device 100 rotates 90 degrees during the execution of the same display processing, the image extension unit 202 generates the extended image data shown in Figure 6B(H) in S304. If the display device 118 of the display device 100 rotates 90 degrees again in this state, the extended image data generation process by S304 is not executed. The display image shown in Figure 6B(E) stored in RAM 114 is displayed again.

[0053] <One aspect of action / effect> Thus, in the second embodiment, the user can arbitrarily select the display method when the aspect ratio of the input image and the display area are different. Furthermore, if the aspect ratio of the display area's resolution changes during the display process, a new image is generated to match the changed aspect ratio, and the input image is expanded. In this way, even if the aspect ratio of the display area changes while the display image is being shown, it is possible to display the image in a way that suppresses a decrease in user satisfaction.

[0054] (Third embodiment) Next, a third embodiment will be described. The third embodiment relates to the operation of the display device 100 when switching display images. The third embodiment also relates to the operation of the display device 100 when setting an area to expand according to the subject of the input image. In the third embodiment, the configuration of the image generation unit 102 and the processing performed therefrom differ from that of the first embodiment, but the other configurations are the same as those of the first embodiment. Therefore, the same reference numerals are used for components and processes that are identical or substantially identical to those described above in the first embodiment, and their descriptions are omitted.

[0055] <Example of the functional configuration of the display device 100> The functional blocks of the image generation unit 102 in the third embodiment will be explained using Figure 7. The image generation unit 102 is composed of a resolution conversion unit 201, an image output unit 203, a subject detection unit 701, and an image expansion unit 702.

[0056] The subject detection unit 701 performs subject detection on the input image data. The subject detection method involves, for example, inputting the input image data into a known deep learning-based image recognition model to detect subject information. Subject information here refers to, for example, people, animals (dogs, cats, birds, horses, etc.), vehicles (cars, motorcycles, trains, etc.), food, landscapes, or buildings. Furthermore, subject information may be detected using training data that enables detection of individuals or specific entities within people or animals, for example. Note that this machine learning-based subject information detection is just one example, and subject information may also be detected by other methods, such as methods that follow predetermined rules.

[0057] The image extension unit 702 uses an image generation AI model to generate a new image that is continuous or consistent with respect to the area surrounding the input image output from the resolution conversion unit 201, thereby extending the input image. At this time, the image extension unit 702 uses subject information output from the subject detection unit 701 to limit the area to be extended within the area surrounding the input image. The image generation AI model is the same as that described in the first embodiment. The image output unit 203 outputs the image data output from the resolution conversion unit 201 or the image extension unit 702 to VRAM.

[0058] <Operation during display processing> The display processing of the display device 100 according to the third embodiment will be explained using Figures 8 and 9. In Figure 8, steps that perform the same operations as in the first embodiment are given the same reference numerals and their explanations are omitted. Furthermore, the operation of each step in the flowchart of Figure 8 is realized by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as necessary. In addition, the display operation by the display device 100 is performed in response to user instructions via the input device 117, as in the first embodiment.

[0059] In S801, the image input unit 101 acquires input image data. The image input unit 101 then outputs the acquired input image data and the aforementioned supplementary information to the image generation unit 102. The image input unit 101 may also process the input image data to make it suitable for use by the image generation unit 102 before outputting it.

[0060] In S802, the resolution conversion unit 201 determines whether the display image data generated from the input image data has already been stored in RAM. If the resolution conversion unit 201 determines that the display image data has been stored, the process proceeds to S305; otherwise, the process proceeds to S302. S302 is the same as in the first embodiment, so its explanation is omitted. After the execution of S302, the process proceeds to S803.

[0061] In S803, the resolution conversion unit 201 determines whether the resolution of the input image acquired by the image input unit 101 is different from the aspect ratio of the display area. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S804; otherwise, the process proceeds to S305. In S804, the subject detection unit 701 detects and acquires subject information from the input image data.

[0062] In S805, the image expansion unit 702 sets an area around the input image to expand it by generating a new image, according to the subject information acquired in S804. In the third embodiment, for example, if a person detected as a subject is cut off at the edge of the input image, the expansion area is not set around the person present in the input image, but the expansion area is set around the input image on the opposite side. The method of setting the area is not limited to the method described above, and the area may be set according to the detection result of a subject of a different type than a person. Also, for example, if the input image cannot be expanded in either the up or down direction, the area where the input image does not exist may be blacked out by the process described in the second embodiment.

[0063] Figures 9(A) to 9(D) illustrate the area where a new image is generated. Figure 9(A) shows an example of an input image acquired by the image input unit 101 in S801. The aspect ratio of the input image is 16:9. In S804, the subject detection unit 701 detects and acquires subject information such as a dog, a person photographing the dog, and background buildings from the input image in Figure 9(A). On the other hand, Figure 9(B) shows an example of the display area of ​​the display device 118, which has an aspect ratio of 3:2.

[0064] Figure 9(C) illustrates a state in which the input image in Figure 9(A) is placed in the display area of ​​Figure 9(B). When the aspect ratio of the input image and the display area are different in this way, the processing by the resolution conversion unit 201 in S302 results in an area at the edge of the display area where the input image does not exist. In this case, the person in the input image of Figure 9(A) is cut off at the bottom of the image. Therefore, in S805, the image expansion unit 702 does not expand the image downwards, but positions the input image to align with the bottom edge of the display area so that the image can be expanded upwards. When the input image is positioned in this way, as in Figure 9(C), an area 901 where the input image does not exist exists on the upper side of the display area. Then, in the processing of S806, a new image is generated for this area 901, and the expanded image is generated.

[0065] More specifically, in S806, the image expansion unit 702 generates a new image with continuity or consistency for the region 901 set in S805, thereby generating a display image. This makes it possible to display the image across the entire display area even when the aspect ratio of the input image and the display area are different. Figure 9(D) shows an example of how the input image in Figure 9(A) is expanded and displayed across the entire display area of ​​Figure 9(B). The process then proceeds to S305.

[0066] S305 and S306 are the same as in the first embodiment, so their explanation is omitted. After the execution of S306, the process proceeds to S807. In S807, the display unit 103 determines whether the display state of the image data has been canceled, that is, whether the image data display process has ended. If it is determined that the image data display process has ended, the process ends; otherwise, the process proceeds to S808. The conditions under which the display state of the image data is canceled are, for example, when the display device 118 is turned off, or when the display state of the display device 118 transitions to another mode. Another mode is, for example, a mode in which a list of thumbnail images is displayed, or a mode in which a settings screen for configuring the display device 100 is displayed.

[0067] In S808, the display unit 103 (an example of the "third determination unit") determines whether or not an instruction has been given to display another image while an image is being displayed. If the display unit 103 determines that an instruction has been given to display another image, the process proceeds to S809; otherwise, the process returns to S807. In S809, the image output unit 203 temporarily stores the display image data being displayed in the RAM 114 (an example of the "second storage unit"). Here, the display image data is stored in the RAM 114 solely for display purposes, and the input image data is not stored in the recording medium 115. Also, if the display image data, including the extended portion, is displayed across the entire display unit 103, the update of supplementary information such as image size is not performed. Then, the process returns to S801.

[0068] By performing the steps described above, for example, if an instruction is given to display the input image of Figure 9(A), and the display image data of Figure 9(D) is generated and displayed on the display unit 103, then an instruction to display another input image is given, and the following process is executed. That is, the extended image of Figure 9(D) is temporarily stored in the RAM 114, and the display image data corresponding to the other input image is displayed on the display unit 103. If an instruction is given to display the image of Figure 9(A) again from that state, the extended image data of Figure 9(D) stored in the RAM 114 is read out as the display image data and displayed on the display unit 103. In other words, if extended image data is temporarily stored in the RAM 114, this extended image data is displayed, and no new extended image data is generated.

[0069] Furthermore, when a user swipes their finger across the touch panel display unit 103 to display images that are earlier or later than the currently displayed image, the next image that can be displayed is limited. Therefore, between the time the display image data is displayed in S306 and the time the display of another image is instructed in S808, processing equivalent to S302 to S806 may be performed on the next image that can be displayed. In this case, an image with a changed resolution (an example of a "fifth image") is generated, and an extended image (an example of a "third extended image") is generated for this image. Display image data may be generated in advance through such processing and stored in RAM 114. In this case, when the display of the next image is instructed in S808, processing S809, S801, and S802 are executed in that order, and then processing S302 to S806 can be omitted, and the process can proceed to S305. Thus, the time until the next image is displayed on the display unit 103 can be shortened.

[0070] <One aspect of action / effect> According to the display device 100 of the third embodiment, when the aspect ratio of the input image and the display area are different, a region 901 is set to expand the image according to the detection result of a subject in the input image. Then, a new image with continuity or consistency is generated for region 901 to expand the input image. As a result, it becomes possible to display the image across the entire display area regardless of the difference in aspect ratio between the input image and the display area.

[0071] Furthermore, according to the display device 100 of the third embodiment, when the display switches to another image during image display, the display image data being displayed is temporarily stored in the RAM 114. This prevents the expanded image content in area 901 from being different each time it is displayed, even if an image is displayed again after another image has been displayed in a series of display processes, by regenerating the image. Therefore, it becomes possible to display the same image that was initially displayed again after another image.

[0072] (Fourth embodiment) Next, a fourth embodiment will be described. In the fourth embodiment, when the aspect ratio of the display area and the input image are different, a predetermined condition is used to determine whether or not to generate an image for the margin area of ​​the display area to expand the input image. In the fourth embodiment, the configuration of the image generation unit 102 is the same as in the first embodiment. Therefore, the same reference numbers are used for components and processes that are the same as or substantially the same as those described in the first embodiment, and their descriptions are omitted.

[0073] <Operation during display processing> The display processing of the display device 100 according to the fourth embodiment will be explained using Figures 10 and 11. In Figure 10, the same reference numerals are used for steps that perform the same operations as in the first embodiment, and their explanations are omitted. Furthermore, the operation of each step in the flowchart of Figure 10 is realized by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as necessary. In addition, the display operation by the display device 100 is performed in response to user instructions via the input device 117, as in the first embodiment.

[0074] Since steps S301 and S302 are the same as in the first embodiment, their explanation is omitted. After the execution of S302, the process proceeds to S1001. In S1001, the resolution conversion unit 201 determines whether the aspect ratio of the input image acquired by the image input unit 101 is different from that of the display area. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S1002; otherwise, the process proceeds to S305.

[0075] In S1002, the image input unit 101 determines whether or not the input image data has a generation extension permission flag attached. If the image input unit 101 determines that the generation extension permission flag is attached, the process proceeds to S1003; otherwise, the process proceeds to S1005. Here, the generation extension permission flag is information indicating whether or not to allow the generation of an image for the blank area that occurs when the aspect ratio of the input image and the display area are different, and to allow the expansion of the input image. Such a generation extension permission flag is embedded in the image file along with the input image data as supplementary information to the input image data.

[0076] In the fourth embodiment, an example of generating an extended image in S1005 when the generation extension permission flag is not set will be described later, but it is not necessary to generate an extended image when the generation extension permission flag is not set. In such a case, the process may proceed to S1004 after S1002 is executed.

[0077] In S1003, the image input unit 101 determines whether the state of the state generation extension permission flag attached to the image is permitted or not. If the image input unit 101 determines that the state is permitted, the process proceeds to S1005; otherwise, the process proceeds to S1004. For example, the input image data is assumed to be an image from a photobook sold as an e-commerce product. In such a case, the publisher sells the product with the generation extension permission flag set to "not permitted" so that the author's and the work's intentions do not change before or after purchase. Note that the setting of the generation extension permission flag may be changed depending on the user's situation, such as before or after purchase, depending on the input image data.

[0078] In S1004, the image output unit 203 adds a black image to display areas where no input image exists. Figure 11(A) illustrates an input image with an aspect ratio of 9:16 acquired by the image input unit 101. Figure 11(B) illustrates the display area of ​​the display device 118 with an aspect ratio of 3:2. Figure 11(C) illustrates a display image when the vertical width of the input image in Figure 11(A) is matched to the vertical width of the display area in Figure 11(B). In the case of Figure 11(C), areas (1101, 1102) where no image exists are created on the left and right sides of the display area, and a black image is added to these areas (1101, 1102).

[0079] In S1005, the image extension unit 202 does not add a black image to the regions (1101, 1102) shown in Figure 11(C). That is, the image extension unit 202 extends the input image by generating a new image for regions (1101, 1102) that is continuous or consistent with the input image. This makes it possible to display the image across the entire display area when the aspect ratio of the input image and the display area are different. Figure 11(F) illustrates an extended image in which the input image in Figure 11(A) is extended across the entire display area of ​​Figure 11(B).

[0080] Furthermore, even when the generation and expansion permission flag is enabled, use cases can be considered where the degree of generation and expansion is limited to prevent significant alteration of the input image. In such cases, the display device program may include a threshold that uniformly limits the degree to which the input image can be generated and expanded. Alternatively, a threshold may be set for each input image by adding metadata to the input image data indicating that the expansion rate is limited.

[0081] For example, region 1107 in Figure 11(D) illustrates the region where the input image, whose resolution has been changed by processing by the resolution conversion unit 201 in S302, is displayed. The expanded regions (1105, 1106) indicate the regions where the input image can be expanded. These regions are calculated using a threshold that limits the expansion ratio. In this way, the expansion region is limited, resulting in regions (1103, 1104) around which there is no input image.

[0082] In such cases, the input image is expanded by generating a new image in the extended region (1105, 1106) that is continuous or consistent with the input image. On the other hand, a black image is applied to the areas of region (1103, 1104) where the extended region (1105, 1106) is not provided. Figure 11(E) shows an example of a display image generated in this way. Note that region (1103, 1104) is an example of a "blank area". Also, region (1105, 1106) is an example of a "predetermined proportion of the blank area". Furthermore, the areas of region (1103, 1104) where the extended region (1105, 1106) is not provided is an example of a "blank area other than the predetermined proportion".

[0083] Alternatively, the prompt, which is text information, may be embedded in the image file along with the input image data, and an image may be generated in the extended area that represents the content written in the prompt. For example, the extended area may be used as an advertising medium by naturally generating an image of the product logo, sign, or the product itself that is written in the prompt. In addition, the input image may be extended by generating a new image with more continuity or consistency by incorporating information about the surrounding circumstances when the image was taken into the prompt. The process then proceeds to S305. S305 and S306 are the same as in the first embodiment, so their explanation is omitted. The process then ends after the execution of S306. The prompt is an example of "data attached to the data of the acquired image".

[0084] <One aspect of action / effect> According to the display device 100 of the fourth embodiment, the control regarding the expansion of the input image is changed according to the information attached to the image. In this way, when expanding the input image and the aspect ratio of the input image and the display area are different, a new image with continuity or consistency with the input image is generated for areas in the display area where the input image does not exist, thereby expanding the input image. This makes it possible to display the image across the entire display area regardless of the difference in aspect ratio between the input image and the display area. Furthermore, according to the display device 100 of the fourth embodiment, conditions are set for when the input image is not expanded. Alternatively, even when expansion is performed, restrictions are placed on the expanded area. In this way, it is possible to protect against alteration of the author's or the intent of the work, or to expand the input image by generating a new image with more continuity or consistency, by allowing the user to specify the content of the expansion.

[0085] (Fifth embodiment) Next, a fifth embodiment will be described. The details described in the first embodiment will be omitted. In the fifth embodiment, the display device 100 according to the first embodiment performs the process of generating an extended image when the operation of the display device 100 is set to a predetermined mode. That is, an extended image is not generated when the operation of the display device 100 is not set to a predetermined mode.

[0086] Below, an editing mode for editing images is given as an example of a predetermined mode. Figure 12(A) illustrates the display screen of the editing mode according to the fifth embodiment. This display screen includes a frame 1201 for cropping the subject and an icon 1202 for switching the screen to the playback mode screen. On the other hand, Figure 12(B) illustrates the display screen of the playback mode according to the fifth embodiment. The playback mode display screen includes an icon 1203 for switching the screen to the editing mode screen.

[0087] The editing mode includes a function to crop images to an arbitrary angle, and cropping is performed by the user manipulating frame 1201. Furthermore, the editing mode and playback mode can be switched between by user operation. Specifically, the transition from editing mode to playback mode is performed by manipulating icon 1202, and conversely, the transition from playback mode to editing mode is achieved by manipulating icon 1203.

[0088] The display processing of the display device 100 according to the fifth embodiment will be described below with reference to Figures 13 to 15. In the flowcharts of Figures 14 and 15, the same reference numerals are used for steps that perform the same operations as in the first embodiment described above, and their explanations are omitted.

[0089] The overall flowchart of the display process according to the fifth embodiment will be explained using Figure 13. Note that the operation of each step in the flowchart of Figure 13 is realized by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as necessary.

[0090] In S1301, the CPU 112 (an example of the "fourth determination unit") determines whether the current operating mode is editing mode. If the CPU 112 determines that the operating mode is editing mode, the process proceeds to S1302; otherwise, the process proceeds to S1303. In S1302, the CPU 112 executes the display process in editing mode. Then the process ends. In S1303, the CPU 112 executes the display process in playback mode. Then the process ends.

[0091] Figure 14 illustrates the flowchart of the detailed operation of the display process in editing mode. Note that the operation of each step in the flowchart in Figure 14 is achieved by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as needed.

[0092] In the flowchart shown in Figure 14, the same process as in the flowchart in Figure 3 of the first embodiment is performed. That is, a new image with continuity or consistency with the input image is generated for display areas where the input image does not exist, which occurs because the aspect ratio of the input image and the display area are different. Then, the display image expanded by this image is displayed. Figure 12(A) illustrates the expanded image generated in this way. The editing mode allows cropping within the range including the expanded image generated in this way. Such a display device 100 can increase the degree of freedom of editing. Note that steps S301 to S306 shown in Figure 14 are the same as the steps in the flowchart shown in Figure 3 of the first embodiment, so their explanation is omitted.

[0093] On the other hand, Figure 15 illustrates a flowchart of the detailed operation of the display processing in playback mode. Note that the operation of each step in the flowchart of Figure 15 is realized by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as needed.

[0094] In the display process shown in Figure 15, a black image is added to the portion of the display area where the input image does not exist, which occurs because the aspect ratio of the input image and the display area are different.

[0095] Steps S301 to S302 are the same as in the first embodiment, so their explanation is omitted. After the execution of S302, the process proceeds to S1501. In S1501, the resolution conversion unit 201 determines whether the aspect ratio of the input image acquired by the image input unit 101 is different from that of the display area. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S1502; otherwise, the process proceeds to S305.

[0096] In S1502, the image output unit 203 adds a black image to the areas (401, 402) located above and below the input image shown in Figure 4(C). As a result, the display image shown in Figure 12(B) is displayed. The process then proceeds to S305. Since S305 to S306 are the same as in the first embodiment, their explanation is omitted. After the execution of S306, the process ends.

[0097] <One aspect of action / effect> According to the display device 100 of this fifth embodiment, the processing of the first embodiment is performed when the device is in editing mode. Therefore, in editing mode, it becomes possible to trim the image to include the generated extended image, thereby increasing the degree of editing freedom. On the other hand, in playback mode, the user does not intend to generate an extended image, and the process of generating an extended image is not executed. In this way, the process of generating an extended image is limited to editing mode, where the user spontaneously edits the image. Therefore, the unnecessary generation of extended images can be prevented.

[0098] (Sixth Embodiment) Next, the display device 100 according to the sixth embodiment will be described. In the sixth embodiment, when the aspect ratio of the display area is changed during image display, the operation of the display device 100 that performs an animation display that smoothly connects from before to after the aspect ratio change will be illustrated. In the sixth embodiment, the configuration of the image generation unit 102 and the processing therefor differ from those of the first embodiment, but other processing is the same as in the first embodiment. Therefore, the same reference numerals are used for components and processes that are the same as or substantially the same as those described above in the first embodiment, and their descriptions are omitted.

[0099] <Example of the functional configuration of the display device 100> The functional blocks of the image generation unit 102 according to the sixth embodiment will be explained using Figure 16. The image generation unit 102 is composed of a resolution conversion unit 1601, an image expansion unit 1602, an image output unit 1603, and an image storage unit 1604.

[0100] The resolution conversion unit 1601 converts the horizontal or vertical dimensions of the input image to the horizontal or vertical dimensions of the display area while maintaining the aspect ratio of the input image's resolution. Alternatively, it converts the input image in accordance with changes in the display area of ​​the display device 100. Changes in the display area of ​​the display device 100 include, for example, rotating the display device 100 by an arbitrary angle.

[0101] The image expansion unit 1602 uses an image generation AI model to generate a new image that is continuous or consistent with respect to the area surrounding the input image output from the resolution conversion unit 1601, thereby expanding the input image. At this time, the image expansion unit 1602 determines the amount of expansion considering the amount of change in the aspect ratio of the display area. The image generation AI model is the same as that described in the first embodiment.

[0102] The image output unit 1603 outputs the image data output from the resolution conversion unit 1601 or the image extension unit 1602 to the image storage unit 1604 (an example of a "third storage unit"). The image storage unit 1604 then stores this image data. The image storage unit 1604 is, for example, the VRAM of RAM 114.

[0103] <Operation during display processing> An example of the display processing of the display device 100 according to the sixth embodiment will be explained using Figures 17 and 18. Hereafter, the transition of the image displayed in the display area will be explained when the display device 118 is rotated 90 degrees clockwise while the display image is displayed in a horizontally elongated display area as shown in Figure 18(C).

[0104] Figure 18(A) illustrates an input image acquired by the image input unit 101. The aspect ratio of the input image is, for example, 16:9. Figure 18(B) illustrates the display area of ​​the display device 118. The aspect ratio of the display area is, for example, 3:2. Figure 18(C) illustrates a display image generated from the input image by the image generation process according to the first embodiment to fit the display area shown in Figure 18(B). On the other hand, Figure 18(D) illustrates the display area when the display device 118 is rotated 90 degrees clockwise.

[0105] The display device 100 according to the sixth embodiment has a function to automatically switch the screen display between portrait and landscape orientation depending on the orientation of the display device 118. Therefore, when the orientation of the display device 118 is rotated by 90 degrees, the aspect ratio of the display area is reversed. For example, the aspect ratio of the width to height in Figure 18(B) is 3:2, while the aspect ratio of Figure 18(D) after a 90-degree rotation is 2:3. Note that the display area in Figure 18(D) is an example of the "third display area".

[0106] Figures 18(E) to 18(H) illustrate, as a comparative example, the display transition when the display device 100 is rotated 90 degrees from the display state in Figure 18(C) without using the method according to the sixth embodiment. When the display device 100 is rotated from landscape to portrait, the aspect ratio of the display area is inverted as shown in Figure 18(E). However, immediately after the inversion, the orientation of the subject included in the display image in Figure 18(C) does not match the orientation of the inverted display area as shown in Figure 18(E). Subsequently, an animation is displayed to show how to align the orientation of the rotated display area with the orientation of the image. That is, the screen of the display device 118 transitions in the order of Figure 18(F) and Figure 18(G).

[0107] Here, in order to match the horizontal width of the display image with the horizontal width of the display area after the aspect ratio has been changed, the width dimension of the display image is gradually reduced from the point in Figure 18(E) to Figures 18(F), 18(G), and 18(H). Also, the display image shown in Figure 18(E) is displayed with a portion that does not fit within the display area cropped out during the transition. When the image transition is complete, the orientation of the image displayed on the screen is finally aligned with the orientation of the display area after the rotation of the subject, as shown in Figure 18(H). Finally, the extended generation process according to the second embodiment is executed on the image in Figure 18(H) to match the aspect ratio after the display area has been changed. Figure 18(I) shows an example of a display image after such processing has been performed.

[0108] In this comparative example, the image generation process is performed so that the aspect ratio of the input image matches the aspect ratio of the display area in Figure 18(C). Furthermore, after that, the extended generation process is performed after the aspect ratio has been changed, as shown in Figure 18(I). However, in this comparative example, during the display of the animation corresponding to the change in the aspect ratio of the display area, areas where there is no display image appear in the corners of the display area. For example, area 1801 in Figure 18(F) and area 1802 in Figure 18(G) are examples of areas where there is no display image. Figures 18(F) and 18(G) illustrate the display area with such areas 1801 and 1802 filled in black.

[0109] Next, a flowchart of the display operation of the display device 100 according to the sixth embodiment will be explained using Figure 17. In Figure 17, steps that perform the same operations as in the first embodiment are given the same reference numerals and their explanations are omitted. Furthermore, the operation of each step in the flowchart of Figure 17 is realized by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as necessary.

[0110] The process in S301 is the same as the process in S301 in Figure 3 of the first embodiment, so its explanation is omitted. The process then proceeds to S1701. In S1701, the resolution conversion unit 1601 of the image generation unit 102 acquires the resolution of the input image and the display area, respectively. The resolution conversion unit 1601 then converts the resolution of the input image data so that the horizontal or vertical width of the input image matches the horizontal or vertical width of the display area, while maintaining the aspect ratio of the input image acquired in S301. The image after the resolution conversion in this way is an example of the "second image".

[0111] In S1702, the image expansion unit 1602 of the image generation unit 102 obtains the resolution of the input image and the display area, and the aspect ratio variation amount of the display area, and determines the expansion amount of the input image. The expansion amount of the input image is determined so that no areas of the display area are not displayed during the animation in which the display image before the aspect ratio of the display area is changed is transitioned to a display image corresponding to the display area after the aspect ratio has been changed.

[0112] The details of S1702 will be explained using Figure 18(J). Figure 18(J) illustrates an extended image obtained by expanding the input image of Figure 18(A) with the image generation process according to the sixth embodiment. More specifically, the rectangular region 1803 illustrates the region of the input image of Figure 18(A). The rectangular region 1804 illustrates the region of the display image generated by executing the image generation process according to the first embodiment. That is, the rectangular region 1804 corresponds to the display region of Figure 18(C).

[0113] On the other hand, the rectangular region 1805 illustrates the region of the image generated when the image generation process according to the second embodiment is executed in accordance with the modified display region, assuming that the aspect ratio of the display region has been changed. That is, the rectangular region 1805 corresponds to the display region in Figure 18(I). Region 1806 illustrates the circle inscribed in this rectangular region 1805.

[0114] Here, we assume that the input image in Figure 18(A) is displayed as a display image with its horizontal width matching the display area in Figure 18(B) while maintaining its aspect ratio (i.e., Figure 18(C)), and that the display area is rotated by 90 degrees as shown in Figure 18(D). In this case, the image inside area 1806 may be displayed on the screen during animation display. Therefore, in S1702, the amount of expansion of the input image is determined so that the input image can be expanded to at least fill area 1806.

[0115] In the sixth embodiment, an example is given of displaying an animation in which the displayed image rotates and transitions when the display area is rotated. Furthermore, in the sixth embodiment, an example of enlarging or shrinking the image displayed on the screen in such a case is described. In such a case, while maintaining the aspect ratio of the image, it is possible to display the image so that its horizontal or vertical width matches that of the display area after the aspect ratio has been changed.

[0116] The transition image may be displayed without changing its magnification while it is rotating. Assuming this condition, the amount of expansion may be determined to expand the input image until it fills an area that includes at least a circle inscribed in the display area that was displayed in the display area before the change.

[0117] Furthermore, the sixth embodiment illustrates a case where the aspect ratio of the display area is changed by rotation of the display device, but the change in the aspect ratio of the display area may be achieved by arbitrary user operation. For example, the image expansion unit 1602 (an example of a "reception unit") may accept user operation to arbitrarily enlarge or reduce the display area using the input device 117. The image expansion unit 1602 may then determine the amount of expansion of the input image as the amount to expand the input image, assuming that the input image is expanded to the area formed by the maximum horizontal width and maximum vertical width that the display area can take. Moreover, the image expansion unit 1602 may determine the amount of expansion of the input image as the amount that allows the input image to be expanded to fill the area of ​​a circle inscribed with the longer of the maximum horizontal width and maximum vertical width that the display area can take.

[0118] Returning to the explanation of the flowchart in Figure 17, in S1703, the image expansion unit 1602 generates a new image that is continuous or consistent with the input image using the expansion amount determined in S1702. Then, the image expansion unit 1602 synthesizes the input image data and the newly generated image data to generate data for an expanded image (an example of a "fourth expanded image").

[0119] The image data generated in S1703 only needs to include at least the region using the expansion amount determined in S1702. For example, if the expansion region determined in S1702 is circular, the expanded image may be generated to fill a rectangular region in which that circle is inscribed.

[0120] Furthermore, in consideration of the processing time required for image generation, the image data generated in S1703 may be generated in such a way that the image quality is reduced for areas that are only displayed during the transition display. For example, in the case of Figure 18(J), even if the area is inside area 1806, the area outside the rectangular area 1805 is temporarily displayed during the animation display, but is not displayed in the final transition image. Therefore, the image data may be generated in such a way that the image quality of the image displayed in the area outside the rectangular area 1805 (an example of "an area outside the predetermined range") is lower than that of the rectangular area 1805 (an example of "a predetermined range").

[0121] In S1704, the image output unit 1603 outputs the extended image data output by the image extension unit 1602 in S1703 to the image storage unit 1604 (an example of the "third storage unit"). The image output unit 1603 also outputs display image data, which is a cropped portion of the extended image data to be displayed, to the image storage unit 1604. The image storage unit 1604 then stores this image data.

[0122] In S1705, the display unit 103 reads the display image data stored in the VRAM of RAM 114 and displays it on the display device 118. In S1706, the display unit 103 determines whether the display state of the display image has been cleared, that is, whether the display process has ended. If the display unit 103 determines that the display process has ended, the process ends; otherwise, the process proceeds to S1707. The conditions under which the display state of the display image is cleared include, for example, when the display device 118 is turned off, or when the display state of the display device 118 transitions to another mode. Another mode may be, for example, a mode in which a list of thumbnail images is displayed, or a mode in which a settings screen for configuring the display device 100 is displayed.

[0123] In S1707, the display unit 103 (an example of the "fifth determination unit") determines whether the aspect ratio of the display area has changed while the display image is being displayed. If the display unit 103 determines that the aspect ratio has changed, the process proceeds to S1708; otherwise, the process returns to S1706.

[0124] In S1708, the resolution conversion unit 1601 acquires the extended image data stored in the image storage unit 1604 in S1704 and converts it into transition image data corresponding to the change in the display area.

[0125] More specifically, the resolution conversion unit 1601 rotates region 1806 in the pre-generated extended image data, as shown in Figure 18(J), 90 degrees clockwise to match the display area with a changed aspect ratio. Then, the resolution conversion unit 1601 rotates region 1806 in the opposite direction by a predetermined angle smaller than 90 degrees, for example. The resolution conversion unit 1601 then enlarges the rotated region 1806 while maintaining its aspect ratio. The resolution conversion unit 1601 then crops a transition image containing the subject by fitting a frame corresponding to the display area with a changed aspect ratio onto this enlarged region 1806. The transition image thus generated includes a subject facing diagonally. The degree of enlargement of region 1806 may be set to a value such that multiple transition images result in a smooth animation.

[0126] Similarly, if S1708 is executed again, the resolution conversion unit 1601 rotates region 1806 in the reverse direction by a predetermined angle to generate another transition image. The resolution conversion unit 1601 continues this process of generating transition images until the orientation of the subject included in the transition image matches the orientation of the display region with the changed aspect ratio, that is, until the transition image shown in Figure 18(M) is generated.

[0127] Figures 18(K) to 18(M) illustrate transition images sequentially generated in S1708 using the extended image shown in Figure 18(J). The transition image shown in Figure 18(K) has the same rotation angle and scale as the transition image in Figure 18(F) related to the comparative example. Similarly, Figure 18(L) corresponds to Figure 18(G), and Figure 18(M) corresponds to Figure 18(H). Figure 18(M) illustrates the final frame among multiple transition images. Thus, when the extended image in Figure 18(J) is converted into a transition image generated using the method of the sixth embodiment, the blank areas (1801, 1802) present in Figures 18(F) to 18(H) do not exist in the transition images from Figure 18(K) to Figure 18(M).

[0128] Note that Figure 18(J) may also be generated by expanding the display image displayed in the display area after the aspect ratio has been changed, as shown in Figure 18(E). In this case, the resolution conversion unit 1601 does not need to rotate area 1806 90 degrees clockwise to match the display area with the changed aspect ratio. The resolution conversion unit 1601 also reduces the size of the expanded image by a predetermined angle while maintaining its aspect ratio. Then, the resolution conversion unit 1601 crops the transition image including the subject by fitting a frame corresponding to the display area with the changed aspect ratio onto the reduced area.

[0129] Returning to the explanation of the flowchart in Figure 17, in S1709, the image output unit 1603 outputs the transition image data converted in S1708 to the image storage unit 1604 as display image data (an example of "sequential output"). The image storage unit 1604 then stores this display image data. In S1710, the display unit 103 reads the state transition image data (display image data) stored in the image storage unit 1604 and displays it on the display device 118.

[0130] In S1711, the display unit 103 determines whether the image transition has finished. This determination is made, for example, by determining whether the orientation of the subject included in the transition image matches the orientation of the display area whose aspect ratio has been changed. If the display unit 103 determines that the transition has finished, the process proceeds to S1712; otherwise, the process returns to S1708. In S1712, the display unit 103 continues to display the last displayed transition image. The process then returns to S1706.

[0131] In the sixth embodiment, when an animation display is performed in conjunction with a change in the aspect ratio of the display area, the display of the last transition image continues after the animation display ends. However, this does not apply to the display format after the animation display ends. For example, as in the second embodiment, image data generated by performing the image generation process again according to the aspect ratio after the change in the display area may be displayed. In this case, the image exemplified in Figure 18(I) is used as the image data for display after the animation display ends.

[0132] <One aspect of action / effect> According to the display device 100 of this sixth embodiment, the input image is generated by extending it beyond the display area, taking into account the change in the aspect ratio of the display area. In this way, it is possible to suppress the occurrence of blank space in the display area during the animation that transitions the display images. Therefore, it is possible to suppress a decrease in user satisfaction.

[0133] (modified version) In the above embodiment, the generation and display of the extended image are performed in the display device 100, but the generation and display of the extended image may be performed in different devices. That is, for example, the extended image is generated in an imaging device that can connect to a network. The extended image data is then transmitted via the network to, for example, the user's smartphone (an example of "outputting extended image data"). The extended image is then displayed on the smartphone's display. It is assumed that the aspect ratio information of the display provided in the smartphone has been transmitted to the imaging device in advance. Even with such modifications, the same effects as in the above embodiment can be achieved.

[0134] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0135] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0136] The disclosures herein include the following information processing devices, methods for controlling them, and programs. (Item 1) A conversion unit that converts the first image into a second image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, In the display area, a position control unit controls the position of the second image converted from the first image by the conversion unit, A generation unit generates an extended image of the second image by generating a third image in the margin area of ​​the display area that does not overlap with the second image whose position is controlled by the position control unit, The system comprises an output unit that outputs the data of the extended image generated by the generation unit, Information processing device. (Item 2) The third image includes an image that is continuous or consistent with the second image. The continuous or consistent image includes an image inferred using a neural network from the pattern or features of the second image. The information processing device described in item 1. (Item 3) If the second image includes a person, the position control unit controls the area around the edge of the second image closest to the person and the edge opposite to that edge to form the margin region. An information processing device as described in item 1 or 2. (Item 4) The system further includes a first determination unit that determines whether or not the generation of the extended image is permitted, If the first determination unit determines that the generation of the extended image is permitted, the generation unit generates the extended image. An information processing device described in any one of items 1 to 3. (Item 5) The region in which the generation unit generates the third image is a predetermined proportion of the margin region, If the first determination unit determines that the generation of the extended image is permitted, The generation unit generates the extended image by generating the third image over a predetermined proportion of the region, thereby extending the second image. The output unit outputs the expanded image generated by the generation unit and a black image generated for the area of ​​the margin region other than the predetermined proportion. The information processing device described in item 4. (Item 6) If the first determination unit determines that the generation of the extended image is not permitted, the output unit outputs the second image and the black image generated for the blank area. The information processing device described in item 4. (Item 7) If the first determination unit determines that the generation of the extended image is not permitted, the output unit outputs an image cropped from the first image with the aspect ratio of the display area, and which is the same size as the entire display area. The information processing device described in item 4. (Item 8) A first storage unit for storing image data, The system further includes a second determination unit that determines whether or not the aspect ratio of the display area has been changed, If the second determination unit determines that the aspect ratio of the display area has been changed, The first storage unit stores the data of the extended image in the output state, The conversion unit converts the first image into a fourth image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the second display area whose aspect ratio has been changed. The position control unit controls the position of the fourth image in the second display area. The generation unit generates a second extended image by extending the fourth image by generating an image that is continuous or consistent with the fourth image in the blank area of ​​the second display area that does not overlap with the fourth image whose position is controlled by the position control unit. The output unit outputs the data of the second extended image generated by the generation unit. If the second determination unit determines that the aspect ratio of the display area has been changed back to the original aspect ratio, the output unit outputs the data of the extended image stored in the first storage unit. An information processing device described in any one of items 1 through 7. (Item 9) The output unit highlights and outputs the region where an image that is continuous or consistent with the second image generated by the generation unit is located. An information processing device described in any one of items 1 through 8. (Item 10) A second memory unit for storing image data, The system further includes a third determination unit that determines whether or not an instruction has been given to display an image different from the extended image that has been output, Before the third determination unit makes the determination, The conversion unit converts the other image into a fifth image by changing the resolution. The position control unit controls the position of the fifth image in the display area. The generation unit generates a third extended image by expanding the fifth image, If the third determination unit determines that the display of an image other than the extended image has been instructed, The second storage unit stores the data of the extended image in the output state. The output unit outputs the third extended image generated by the generation unit. The output unit further outputs the extended image stored in the second storage unit in response to an instruction to display the extended image. An information processing device described in any one of items 1 through 9. (Item 11) The information processing device further comprises a fourth determination unit for determining the operating mode of the information processing device, If the fourth determination unit determines that the aforementioned operating mode is an editing mode for editing image data, The output unit outputs the extended image generated by the generation unit. If the fourth determination unit determines that the operation mode is a playback mode for playing back image data, the output unit outputs the second image and the black image generated for the blank area. An information processing device described in any one of items 1 through 10. (Item 12) The output unit generates an image for the blank area using the data attached to the data of the first image. An information processing device described in any one of items 1 through 11. (Item 13) A third memory unit that stores image data, The system further includes a fifth determination unit that determines whether the display area has been changed to a third display area having a different aspect ratio from the display area, Before the determination by the fifth determination unit, The generation unit generates a fourth extended image by generating an image that is continuous or consistent with the second image, thereby extending the second image to a predetermined area larger than the extended image. The third storage unit stores the data of the fourth extended image generated by the generation unit. If the fifth determination unit determines that the display area has been changed to the third display area, The conversion unit converts the data of the fourth extended image stored in the third storage unit into data of a plurality of transition images corresponding to the change from the display area to the third display area. The output unit sequentially outputs the data of the plurality of transition images. The information processing device described in item 1. (Item 14) The predetermined region includes a region of such size that, when each of the plurality of transition images is superimposed on the third display region, no blank area is created in the third display region. The information processing device described in item 13. (Item 15) Changing from the aforementioned display area to the third display area includes maintaining the size of the aforementioned display area and changing the aspect ratio of the aforementioned display area. The predetermined region includes at least a circle in which the display region is inscribed. The information processing device described in item 14. (Item 16) The system further includes a reception unit that receives operations to enlarge or reduce the aforementioned display area. The information processing device described in item 13. (Item 17) The predetermined area includes at least the area formed by the maximum horizontal width and maximum vertical width that the display area received by the receiving unit can take. The information processing device described in item 16. (Item 18) The predetermined area includes at least a circle inscribed within the longer of the maximum horizontal width and maximum vertical width that the display area received by the receiving unit can take. The information processing device described in item 16. (Item 19) The generation of the fourth extended image by the generation unit includes generating the fourth extended image by making the quality of the image of a predetermined range within the predetermined region different from the quality of the image outside the predetermined range within the predetermined region. The information processing device described in item 13. (Item 20) The generation of the fourth extended image by the generation unit includes generating the image quality of the predetermined range at a higher level than the image quality of the image outside the predetermined range. The predetermined range includes the region displayed in the last transition image among the plurality of transition images. The information processing device described in item 19. (Item 21) A method for controlling an information processing device, The conversion unit performs a conversion step of converting the first image into a second image by changing the resolution so that the aspect ratio of the first image is maintained and the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, The position control unit performs a position control step in which it controls the position of the second image converted from the first image in the conversion step within the display area, The generation unit generates a third image in the blank area of ​​the display region that does not overlap with the second image whose position has been controlled in the position control step, thereby generating an extended image of the second image. The output unit includes an output step of outputting the data of the extended image generated in the generation step, A method for controlling an information processing device. (Item 22) A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method is: The conversion unit performs a conversion step of converting the first image into a second image by changing the resolution so that the aspect ratio of the first image is maintained and the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, The position control unit performs a position control step in which it controls the position of the second image converted from the first image in the conversion step within the display area, The generation unit generates a third image in the blank area of ​​the display region that does not overlap with the second image whose position has been controlled in the position control step, thereby generating an extended image of the second image. The output unit includes an output step of outputting the data of the extended image generated in the generation step, program. [Explanation of Symbols]

[0137] 100: Display device, 101: Image input unit, 102: Image generation unit, 103: Display unit, 112: CPU, 113: ROM, 114: RAM, 116: GPU, 117: Input device, 118: Display device, 201: Resolution conversion unit, 202: Image expansion unit, 203: Image output unit

Claims

1. A conversion unit that converts the first image into a second image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, In the display area, a position control unit controls the position of the second image converted from the first image by the conversion unit, A generation unit generates an extended image of the second image by generating a third image in the margin area of ​​the display area that does not overlap with the second image whose position is controlled by the position control unit, The system comprises an output unit that outputs the data of the extended image generated by the generation unit, Information processing device.

2. The third image includes an image that is continuous or consistent with the second image. The continuous or consistent image includes an image inferred using a neural network from the pattern or features of the second image. The information processing apparatus according to claim 1.

3. The position control unit, when the second image includes a person, controls the area around the edge of the second image closest to the person and the opposite edge to that edge to form the margin region. The information processing apparatus according to claim 1.

4. The system further includes a first determination unit that determines whether or not the generation of the extended image is permitted, If the first determination unit determines that the generation of the extended image is permitted, the generation unit generates the extended image. The information processing apparatus according to claim 1.

5. The region in which the generation unit generates the third image is a predetermined proportion of the margin region, If the first determination unit determines that the generation of the extended image is permitted, The generation unit generates the extended image by generating the third image over a predetermined proportion of the region, thereby extending the second image. The output unit outputs the expanded image generated by the generation unit and a black image generated for the area of ​​the margin region other than the predetermined proportion. The information processing apparatus according to claim 4.

6. If the first determination unit determines that the generation of the extended image is not permitted, the output unit outputs the second image and the black image generated for the blank area. The information processing apparatus according to claim 4.

7. If the first determination unit determines that the generation of the extended image is not permitted, the output unit outputs an image cropped from the first image with the aspect ratio of the display area, which is the same size as the entire display area. The information processing apparatus according to claim 4.

8. A first storage unit for storing image data, The system further includes a second determination unit that determines whether or not the aspect ratio of the display area has been changed, If the second determination unit determines that the aspect ratio of the display area has been changed, The first storage unit stores the data of the extended image in the output state, The conversion unit converts the first image into a fourth image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the second display area whose aspect ratio has been changed. The position control unit controls the position of the fourth image in the second display area. The generation unit generates a second extended image by extending the fourth image by generating an image that is continuous or consistent with the fourth image in the blank area of ​​the second display area that does not overlap with the fourth image whose position is controlled by the position control unit. The output unit outputs the data of the second extended image generated by the generation unit. If the second determination unit determines that the aspect ratio of the display area has been changed to the original aspect ratio, the output unit outputs the data of the extended image as stored in the first storage unit. The information processing apparatus according to claim 1.

9. The output unit highlights and outputs the region in which an image that is continuous or consistent with the second image generated by the generation unit is located. The information processing apparatus according to claim 1.

10. A second memory unit for storing image data, The system further includes a third determination unit that determines whether or not an instruction has been given to display an image different from the extended image that has been output, Before the third determination unit makes the determination, The conversion unit converts the other image into a fifth image by changing the resolution. The position control unit controls the position of the fifth image in the display area. The generation unit generates a third extended image by expanding the fifth image, If the third determination unit determines that the display of an image other than the extended image has been instructed, The second storage unit stores the data of the extended image in the output state. The output unit outputs the third extended image generated by the generation unit. The output unit further outputs the extended image stored in the second storage unit in response to an instruction to display the extended image. The information processing apparatus according to claim 1.

11. The information processing device further comprises a fourth determination unit for determining the operating mode, If the fourth determination unit determines that the operation mode is an editing mode for editing image data, The output unit outputs the extended image generated by the generation unit. If the fourth determination unit determines that the operation mode is a playback mode for playing back image data, the output unit outputs the second image and the black image generated for the blank area. The information processing apparatus according to claim 1.

12. The output unit generates an image for the blank area using the data attached to the data of the first image. An information processing apparatus according to any one of claims 1 to 11.

13. A third storage unit for storing image data, The system further includes a fifth determination unit that determines whether the display area has been changed to a third display area having a different aspect ratio from the display area, Before the determination by the fifth determination unit, The generation unit generates a fourth extended image by generating an image that is continuous or consistent with the second image, thereby extending the second image to a predetermined area larger than the extended image. The third storage unit stores the data of the fourth extended image generated by the generation unit. If the fifth determination unit determines that the display area has been changed to the third display area, The conversion unit converts the data of the fourth extended image stored in the third storage unit into data of a plurality of transition images corresponding to the change from the display area to the third display area. The output unit sequentially outputs the data of the plurality of transition images. The information processing apparatus according to claim 1.

14. The predetermined region includes a region of such size that, when each of the plurality of transition images is superimposed on the third display region, no blank area is created in the third display region. The information processing apparatus according to claim 13.

15. Changing from the aforementioned display area to the third display area includes maintaining the size of the aforementioned display area and changing the aspect ratio of the aforementioned display area. The predetermined region includes at least a circle in which the display region is inscribed. The information processing apparatus according to claim 14.

16. The system further includes a reception unit that receives operations to enlarge or reduce the aforementioned display area. The information processing apparatus according to claim 13.

17. The predetermined area includes at least the area formed by the maximum horizontal width and maximum vertical width that the display area received by the receiving unit can take. The information processing apparatus according to claim 16.

18. The predetermined area includes at least a circle inscribed within the longer of the maximum horizontal width and maximum vertical width that the display area received by the receiving unit can take. The information processing apparatus according to claim 16.

19. The generation of the fourth extended image by the generation unit includes generating the fourth extended image by making the quality of the image of a predetermined range within the predetermined region different from the quality of the image outside the predetermined range within the predetermined region. The information processing apparatus according to claim 13.

20. The generation of the fourth extended image by the generation unit includes generating the image quality of the predetermined range at a higher level than the image quality of the image outside the predetermined range. The predetermined range includes the region displayed in the last transition image among the plurality of transition images. The information processing apparatus according to claim 19.

21. A method for controlling an information processing device, The conversion unit performs a conversion step of converting the first image into a second image by changing the resolution so that the aspect ratio of the first image is maintained and the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, The position control unit performs a position control step in which it controls the position of the second image converted from the first image in the conversion step within the display area, The generation unit generates a third image in the blank area of ​​the display region that does not overlap with the second image whose position has been controlled in the position control step, thereby generating an extended image of the second image. The output unit includes an output step of outputting the data of the extended image generated in the generation step, A method for controlling an information processing device.

22. A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method is: The conversion unit performs a conversion step of converting the first image into a second image by changing the resolution so that the aspect ratio of the first image is maintained and the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area, The position control unit performs a position control step in which it controls the position of the second image converted from the first image in the conversion step within the display area, The generation unit generates a third image in the blank area of ​​the display region that does not overlap with the second image whose position has been controlled in the position control step, thereby generating an extended image of the second image. The output unit includes an output step of outputting the data of the extended image generated in the generation step, program.

Images