Image processing device and program

Abstract

The present invention provides an image processing device that can apply image effects to subjects included in images such as landscapes. [Solution] The image processing device includes an image acquisition unit that acquires an image, a subject type identification unit that identifies the type of object which is the type of at least one subject included in the image acquired by the image acquisition unit, an image effect application unit that applies an image effect to the subject in the image, which is set based on the size of the subject in the image and the object type of the subject, and which mimics the way the subject is viewed when viewed through a translucent member, and an image output unit that outputs the image after the image effect has been applied by the image effect application unit as display image data.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and a program used for a display device that displays images.

Background Art

[0002] A window-type display installed on an indoor wall has been devised (see, for example, Non-Patent Document 1). For example, by installing such a window-type display in a room where a window cannot be installed, it is possible to give a feeling of openness as if there is a window there.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, there is a problem that only an image is displayed on the window-type display, and the sense of reality as a window is insufficient.

[0005] In view of the above points, the present invention has been made, and one of the problems is to provide an image processing apparatus that can impart an image effect for supplementing the sense of reality to a subject included in an image such as a landscape.

Means for Solving the Problems

[0006] The invention described in claim 1 is characterized by comprising: an image acquisition unit that acquires an image; a subject type identification unit that identifies the type of object which is the type of at least one subject included in the image acquired by the image acquisition unit; an image effect application unit that applies an image effect to the subject in the image, which is set based on the size of the subject in the image and the object type of the subject, and which mimics the way the subject is viewed when viewed through a translucent member; and an image output unit that outputs the image after the image effect has been applied by the image effect application unit as display image data.

[0007] The invention described in claim 13 comprises: an image acquisition unit that acquires an image; an image effect application unit that applies an image effect to at least one subject included in the image acquired by the image acquisition unit, which mimics the way the subject is viewed when viewed through a patterned glass plate; and an image output unit that outputs the image after the image effect has been applied by the image effect application unit as display image data, wherein the image effect application unit divides the image acquired by the image acquisition unit into a plurality of image regions corresponding to the pattern of the patterned glass plate, and changes the image effect applied to the subject for each image region.

[0008] The invention described in claim 14 is a program that causes a computer to execute an image acquisition step of acquiring an image; a subject type identification step of identifying the type of object which is the type of at least one subject included in the image acquired in the image acquisition step; an image effect application step of applying an image effect to the subject in the image, which is set based on the size of the subject in the image and the object type of the subject, and which mimics the way the subject is viewed when viewed through a translucent member; and an image output step of outputting the image after the image effect has been applied by the image effect application step as display image data. [Brief explanation of the drawing]

[0009] [Figure 1]This is a block diagram showing the overall configuration of a system including an image processing device according to an embodiment of the present invention. [Figure 2] This figure illustrates the patterned glass used in image processing for the display device of the embodiment. [Figure 2A] This figure illustrates the pattern of partitioned regions that mimic the uneven surface areas of patterned glass in the image processing of the display device of the embodiment. [Figure 3] This figure illustrates the further patterned glass used in image processing for the display device of the embodiment. [Figure 4] This figure illustrates further patterns of glass used in image processing for the display device of the embodiment. [Figure 5] This is a flowchart illustrating the basic operation of the image processing device according to this embodiment. [Figure 6] This figure illustrates an example of an image that is the target of image processing related to the display device of the embodiment. [Figure 7] Figure 6 illustrates an example image after image processing to show an image with a blurring effect applied. [Modes for carrying out the invention]

[0010] The following describes embodiments of the image display device according to the present invention with reference to the drawings. However, the present invention is not limited to the following embodiments. Furthermore, in the embodiments, components having substantially the same function and configuration are denoted by the same reference numerals, and redundant explanations are omitted. [Examples]

[0011] Figure 1 shows a block diagram of the overall system configuration including the image processing device 100 of the embodiment. The image processing device 100 can be implemented as a computer device on which an image processing program controlled by a software operating system (OS) is installed.

[0012] In the image processing device 100 shown in Figure 1, the CPU (Central Processing Unit) 111 controls the entire system by using the RAM (Random Access Memory) 113 as primary storage work memory, according to a program stored in a storage unit 112 such as a hard disk drive or SSD (Solid State Drive). Furthermore, the CPU 111 executes image processing according to the embodiment based on the program stored in the storage unit 112, according to user instructions input via a pointing device 114 (or a wireless remote control, mouse, touch panel, etc.) or a keyboard 115.

[0013] The display 116, which includes an LCD (Liquid Crystal Display) or the like and is connected to the image processing device 100, displays an input waiting screen for image processing performed by the CPU 111, processing progress and results, and the image after image effects have been applied. The display 116 may allow the user to directly view the LCD display surface, or it may have an optical element such as a Fresnel lens spaced apart on the front side of the LCD so that the image displayed on the LCD display surface is viewed as an image with a sense of depth. In this embodiment, the image processing device may be configured as a system consisting of a display with a pseudo-actual window frame FL surrounding the display screen and a personal computer. Alternatively, the image processing device may be a personal computer or the like excluding the display.

[0014] The image processing device 100 also includes a USB (Universal Serial Bus) 117 for connecting external devices, such as a recording medium, and a network connection unit (network interface card) 119 for communicating with the outside world. The pointing device 114, keyboard 115, USB 117, and network connection unit 119 function as input units. These components are connected to each other via a predetermined bus.

[0015] The storage unit 112 of the image processing apparatus 100 stores an image processing program 200 that causes the image processing apparatus 100 to function. The CPU 111 reads out the image processing program 200 from the storage unit 112, expands it in the RAM 113 (or the storage unit 112 if necessary), and sequentially executes the processes included in the image processing program 200. The image processing program 200 is a program that causes the image processing apparatus 100 to function as an image data acquisition unit 120, a subject type identification unit 121, an image effect application execution unit 122, an effect condition setting unit 123, and an image output unit 124, which will be described later.

[0016] Further, the storage unit 112 stores, for example, a condition file for subject type identification of the image effect application execution unit 122 and the effect condition setting unit 123 based on the image processing program 200. In the condition file of the storage unit 112, for each type of subject (human / standing tree / building / window / car, etc.) used by the subject type identification unit 121, the corresponding reference size (number of pixels in the vertical and horizontal directions, etc.) is associated and stored. Also, in the storage unit 112, data corresponding to the optical conditions of light-transmitting members such as float glass plates, ground glass plates, wire mesh glass plates, and patterned glass plates, which are necessary for visual effects such as distortion, blurring, and brightness change, is also stored as a visual effect optical condition file.

[0017] [Image Data Acquisition Unit 120] The image data acquisition unit 120 acquires images via input units such as the USB 117 and the network connection unit 119. Typically, the image data is an image of a landscape taken. The image may be a moving image or a still image.

[0018] [Subject Type Identification Unit 121] The subject type identification unit 121 identifies the object type, which is the type of at least one subject included in the image acquired by the image data acquisition unit 120. Subjects include items such as humans, animals, plants (trees, standing trees), vehicles (automobiles), and buildings, which are the classes in the classification in AI image identification.

[0019] The subject type identification unit 121 identifies the object type of the subject in the acquired image and generates the subject distance based on a comparison between the standard size corresponding to the identification result and the size of the subject in the image. Specifically, the subject type identification unit 121 calculates the ratio of the size of the first subject in the acquired image to the standard size for the object type of the first subject, and estimates the subject distance, which is the distance between the imaging device and the first subject at the time of image capture, based on this ratio.

[0020] [Image effect application execution unit 122] The image effect application unit 122 applies an image effect (hereinafter also referred to as the "glass-through effect") to the subject or background in the image that mimics how the subject would appear when viewed through a translucent material. The image effect application unit 122 sets the image effect to be applied to the subject according to the subject distance generated by the subject distance generation unit 121a. That is, the image effect to be applied to the subject is set based on the size of the subject in the image and the object type of the subject.

[0021] The "glass effect" is a visual effect that makes it seem as if you are viewing the subject and background through a glass plate. The "glass effect" includes image effects such as distortion, blurring, and brightness changes that make it seem as if you are viewing the subject through various types of glass plates, such as frosted glass or patterned glass. In applying the blurring image effect, for each of the multiple subjects in the image, a circle of confusion with a different radius corresponding to the distance to the subject is applied to the area corresponding to each subject, thereby changing the blur intensity.

[0022] The following processes (1) to (4) can be listed as image processing methods for each type of glass performed by the image effect application execution unit 122.

[0023] (1) In the case of float glass, the blurring process is applied uniformly and slightly, regardless of the distance to the subject, without applying strong blurring.

[0024] (2) In the case of frosted glass, the blurring process is adjusted according to the distance of the subject, with the blurring becoming stronger the further the subject is from the virtual window.

[0025] (3) In the case of patterned glass as shown in Figure 2, as shown in Figure 2A, partitioned areas that mimic multiple uneven regions on the glass surface are set in advance, and corresponding image processing is performed for each partitioned area. Each partitioned area is classified into several ranks (ranks 1 to 4 in Figure 2A), and a weighting coefficient for blurring is set according to the rank. In addition to the weighting coefficient for blurring, a different luminance attenuation coefficient may be set for each partitioned area. In this way, the image effect application execution unit 122 can divide the image acquired by the image data acquisition unit 120 into multiple partitioned areas corresponding to a predetermined pattern, and change the image effect applied to the subject for each partitioned area. Furthermore, for the boundary parts between partitioned areas, the average value of the effect application of the adjacent partitioned areas may be adopted to ensure that the image changes smoothly. Further patterns for partitioned areas of patterned glass may include, for example, a pattern that mimics a natural object N, as shown in Figure 3. Further other patterns for partitioned areas of patterned glass may include a mosaic pattern (the image of the cat is mosaicked), as shown in Figure 4.

[0026] (4) In the case of wired glass, it may be combined with any of the above types of glass, and image processing is performed to superimpose a mesh-like pattern onto an image that has already been processed, such as blurring, changing brightness, or mosaicing.

[0027] [Effect condition setting unit 123] The effect condition setting unit 123 acquires user input and functions as a glass type selection unit that selects the type of glass plate (float glass / patterned glass / frosted glass / wired glass) to be used for the glass-through effect applied to the image acquired by the image data acquisition unit 120. The user can select and specify the type of virtual glass to be used for the virtual window via the input unit.

[0028] [Image output unit 124] The image output unit 124 displays an image with a predetermined image effect applied to it on the display unit 116. The image output unit 124 may be equipped with at least one of a scaler function that enlarges or reduces the resolution of the display image data to match the resolution of the display 116, and a cropping function that displays the image to match the aspect ratio of the display 116.

[0029] Furthermore, although an LCD is used in the above embodiment, other displays such as an organic electroluminescent display may be used instead of an LCD.

[0030] Alternatively, instead of placing a physical window frame on the display unit's display 116, the image output unit 124 may superimpose a window frame image surrounding the display screen onto the display image data that already has a glass-through effect applied.

[0031] Furthermore, in addition to applying the glass-through effect, other processing may be performed on the original image. That is, the subject type identification unit 121 can be configured to change the background of the subject or add an additional subject in the image based on the attributes of the subject and the background other than the subject. For example, the subject type identification unit 121 can change a cloudy sky to a blue sky, and the subject type identification unit 121 can recognize the scene in the original image through AI processing and perform processing on the original image according to the scene (for example, processing to add an image of a "flying butterfly" to an image of a flower field in the original image).

[0032] [Operation of the image processing device] Figure 5 shows the image processing execution flow of the image processing apparatus in this embodiment.

[0033] When power is turned on to the image processing device 100 and the image processing device 100 starts operating, it enters a standby state where it can accept user input.

[0034] In step S1 (image acquisition step), the image processing device 100 acquires image data input via the USB 117 input unit or the network connection unit 119. That is, the image data acquisition unit 120 of the device acquires image data from a recording medium connected by wire or wireless. The acquired image is then stored in the storage unit 112 and displayed on the display 116 via the image output unit 124.

[0035] In step S2 (effect condition setting step), the effect condition setting unit 123 displays a mode selection menu on the display 116 via the image output unit 124 and prompts the user to select or deselect the "apply image effect" mode from the displayed menu. In this embodiment, there are multiple operation mode selections, but only the mode for applying the effect through glass will be explained, and the explanation of the other modes will be omitted. For example, the effect condition setting unit 123 accepts the user's mode selection to specify the type of glass plate as the effect through glass, such as a float glass plate, frosted glass plate, wired glass plate, or patterned glass plate.

[0036] If the user selection operation in step S2 results in the selection of an image effect mode (Y), the process proceeds to step S3. If no image effect mode is selected (N), the image is maintained (step S6).

[0037] In step S3 (subject type identification step), the subject type identification unit 121 identifies the object type, such as a tree, included in the image data acquisition unit 120, and the subject distance generation unit 121a of the subject type identification unit generates the subject distance of the tree based on the comparison result between the standard size of the tree and the size of the tree in the image, according to the identification result.

[0038] Figure 6 shows an example image in which six trees are captured. The six trees are at different virtual distances (hereinafter also simply referred to as distance) from the virtual window. Trees T1 to T6 are located at distances (T1 to T6) from the virtual window in order, and backgrounds C1 to C3 are also located at distances (C1 to C6) from the virtual window in order.

[0039] In step S4 (image effect application step), based on the virtual distance for each subject generated in step 3, a glass-through effect corresponding to the glass plate type selected in step 2 is applied to the acquired image. For example, if the frosted glass effect was selected in step 2, the image effect application execution unit 122 performs a frosted glass blurring process for each tree T1-T6 and background C1-C6 according to the distances of the trees T1-T6 (T1-T6) and background C1-C6 (C1-C6) obtained in step S3, applies it to the image, and outputs the effect-applied image via the image output unit 124 (image output step). That is, the image output unit 124 outputs the image after the image effect application by the image effect application execution unit 122 as display image data to the display 116.

[0040] Figure 7 shows images with the effect of viewing through frosted glass applied. Since the trees T1 and background C1 at distances T1 and C1 are close, their images appear slightly blurred. Images of trees T2-T6 and backgrounds C2-3, which are further away, become significantly blurred depending on the distance.

[0041] In step S5 (video detection step), it is determined whether the image to which the blurring effect has been applied is a still image or a video. Here, if the acquired image is a video (multiple still images that switch in chronological order) (Y), the process returns to step S3 and continues. In this way, when the image switches in chronological order, the processing in steps S3 to S4 is repeated, allowing the frosted glass effect in the image effect application execution unit 122 to be applied to each frame of the video image.

[0042] In step S6, if the acquired image is a still image (N), the still image is maintained on the display 116.

[0043] According to the image processing device 100 of this embodiment, not only pre-prepared image content but also images desired by the user can be given a glass-through effect and displayed on the display. Furthermore, by installing such an image processing device on a wall in a room, the sense of realism as a window is enhanced, and the user can feel a sense of openness as if there were a window there.

[0044] Furthermore, the configurations of the embodiments described above may be combined or some of the components may be replaced. Also, the configuration of the present invention is not limited to the embodiments described above, and various modifications may be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0045] 100 Image Processing Devices 120 Image data acquisition unit 121 Subject type identification unit 121a Subject distance generator 122 Image effect application execution unit 123 Effect Condition Setting Section 124 Image output section 200 Image Processing Programs

Claims

[Claim 1] Image acquisition unit that acquires images, A subject type identification unit identifies the object type, which is the type of at least one subject included in the image acquired by the image acquisition unit, An image effect application unit applies an image effect to the subject in the image, which is set based on the size of the subject in the image and the object type of the subject, and which mimics the way the subject is viewed when viewed through a translucent member. An image processing apparatus having an image output unit that outputs the image after the image effect has been applied by the image effect application unit as display image data.

Images