Image display device, system and method

Abstract

The present invention provides an image display device and method, which include a display, the display being arranged at a user's preferred position, and displaying an image with a comfortable image quality and arrangement that matches the user's viewing position and preference. The image display device (10) includes an external input interface (7), a memory (4) for storing image data, an arithmetic processing unit (3), a graphics processing unit (2), a display (1), and a setting mode suitable for the user.

Description

Technical Field

[0001] The present disclosure relates to an image display device and method having image reproduction and arrangement functions that conform to user preferences. Background Art

[0002] Conventionally, as an image display device that reproduces a user's preferred image, a video recording and reproducing device has been proposed that allows a user to send a command from a home viewing device and view a preferred movie via a network (see, for example, Patent Document 1).

[0003] In addition, a method is proposed, comprising the following steps: capturing an image of a consumer using a camera; performing a biometric scanning on the image to determine whether a biometric can be identified from the scan; if a biometric is identified from the scan, the system accesses a database of biometrics associated with at least one of a browsing history, a purchase history, and preference data associated with the consumer; and displaying an advertising image consistent with the data stored in the database on a display device (for example, refer to patent document 2).

[0004] As a display device that utilizes tracking of viewer parts such as the face, a stereoscopic image display device is proposed, which causes a display panel to display a viewing area image indicating whether the viewer is within the viewing area based on the distance from the display device to the viewer and the position of the viewer detected by a tracking unit (for example, refer to patent document 3).

[0005] As a method of controlling multiple displays to display images divided into multiple displays, the following method is proposed: a main controller receives a first display state representing the display state of a first portion of an image of the first display from a first display, and receives a second display state representing the display state of a second portion of an image of the second display from a second display (for example, refer to patent document 4).

[0006] Prior Art Literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2019-92152

[0009] Patent Document 2: Japanese Patent No. 5893737

[0010] Patent Document 3: Japanese Patent No. 5050120

[0011] Patent Document 4: Japanese Patent Application Publication No. 2013-546042 Summary of the invention

[0012] -Problems to be solved by the invention-

[0013] An object of the present invention is to provide an image display device and a method, wherein the image display device includes a display that is arranged at a user's preferred position and displays an image with a comfortable image quality and arrangement that matches the user's viewing position and preference.

[0014] -Methods for solving problems-

[0015] In one embodiment, the present disclosure provides an image display device including an external input interface, a memory for storing image data, an arithmetic processing unit, a graphics processing unit, a display, and a setting mode suitable for a user.

[0016] As another embodiment, an image display device and a display method using the device are provided, wherein the image display device comprises: a user data file related to a user output from the image display device, a server connected to the Internet, and an Internet data file of given data obtained from the Internet, wherein the server comprises an inference function, which outputs recommended landscape image data preferred by the user according to a given algorithm based on or based on the input of data from the user data file and the Internet data file.

[0017] In addition, as another embodiment, an image display device and a display method using the device are provided, wherein the image display device comprises: a display for displaying images; a memory for storing images; an Internet interface for reading image data stored in the Internet; a decoding chip; and a video memory, wherein the image data stored in the memory and / or the Internet is read out, and the read image data is decoded by the decoding chip according to the format of the image data, and a frame image generated by the decoding process for each image frame is stored in the video memory, and the frame image stored in the video memory is displayed on the display in the order of the image frames.

[0018] As another embodiment, a display device and a display method using the device are provided, wherein the display device comprises: the display for displaying a desired scenery; a unit for setting spatial information of the scenery; a unit for receiving notifications from the outside; a unit for drawing the notifications from the outside into an object that can be inserted into an image space; and a unit for dynamically inserting the object into the image.

[0019] As another embodiment, an image display device and a display method using the same are provided, wherein the image display device comprises: a camera for identifying a position of a viewer located in front of the display; and a computer graphics generation unit for generating an image of the display, wherein the display image of the display moves in conjunction with the movement of a given part of the viewer.

[0020] As another embodiment, an image display device and a display method using the device are provided, wherein the image display device includes a plurality of displays arranged arbitrarily, an electronic device for acquiring an image including the plurality of displays, a unit for detecting the positions of the plurality of displays in the acquired image, and a unit for clipping an image from the detected positions of the plurality of displays and displaying it on the displays. The clipped image can be synchronously displayed on the displays at the respectively detected positions.

[0021] -Effects of the Invention-

[0022] According to the present disclosure, a display can be arranged at a user's preferred position such as a wall, and images with comfortable image quality and arrangement matching the user's viewing position and preference can be displayed on the display. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a simplified block diagram showing the structure of a landscape image recording and reproducing system according to an embodiment of the present disclosure.

[0024] Figure 2 Yes means Figure 1 An illustration of an operation example of a landscape image recording and reproducing system.

[0025] FIG. 3( a ) is a diagram showing scenery images sequentially displayed on the image recording and reproducing device as an example of this embodiment.

[0026] FIG. 3( b ) is a diagram showing scenery images sequentially displayed on the image recording and reproducing device as another example of the present embodiment.

[0027] FIG. 3( c ) is a diagram showing scenery images sequentially displayed on the image recording and reproducing device as another example of the present embodiment.

[0028] Figure 4 It is a simplified block diagram showing the structure of an image reproduction device according to an embodiment of the present disclosure.

[0029] Figure 5 yes Figure 4 Flow chart of image processing of an image reproduction device.

[0030] Figure 6 yes Figure 4 A flowchart of deformed image processing of an image reproduction device.

[0031] Figure 7 (a) is Figure 4 (b) is also an example of color tone correction of a landscape image corresponding to the content of the landscape displayed on the display.

[0032] Figure 8 (a) is related to Figure 4 (b) is an example of image correction corresponding to the installation environment of the image reproduction device, and (b) is an example of applying a given filtering process to a landscape image.

[0033] Fig. 9 The following are examples of adding spatial information to an image displayed on a display representing an embodiment of the present disclosure. (a) is an original image, (b) is an image to which spatial information is added, and (c) is an image to which new information is added.

[0034] Fig.10 yes Fig. 9 Examples of images of external notifications of incoming mail on a display, where (a) is the original image and (b) is the image when there is an incoming mail.

[0035] Fig.11 (a) and (b) are front views of a display that displays an image based on the position of a person in front of an image display device representing an embodiment of the present disclosure, and (c) and (d) are camera views of the person.

[0036] Fig.12 (a) and (b) represent Fig.11 Computer graphics illustration of the camera configuration.

[0037] Fig.13 As one embodiment of the present disclosure, (a) is an explanatory diagram of cropping display using position detection of a plurality of displays, and (b) is a diagram showing an image cropped in (a) being displayed on a plurality of displays.

[0038] Fig.14 As an example, (a) is an explanatory diagram of a method of detecting the relative distance between a plurality of displays, and (b) is an explanatory diagram of another method of detecting the relative distance between displays.

[0039] Fig.15 As an example, (a) is a diagram showing an initial arrangement state of a plurality of displays, and (b) is an explanatory diagram of detection of movement of the displays.

[0040] Fig.16 As an example, (a) is an explanatory diagram of a change in the magnification of a cropped image, an explanatory diagram of a method of detecting relative distances between a plurality of displays, and (b) is an explanatory diagram of a leader display and a follower display. DETAILED DESCRIPTION

[0041] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the following description, the same reference numerals are given to the same components and components. The present disclosure includes, for example, the following structures.

[0042] [Structure 1]

[0043] An image display device comprises: an external input interface (7); a memory (4) for storing image data; an arithmetic processing unit (3); a graphics processing unit (2); a display (1); and a setting mode suitable for a user. The device can include a setting for image quality suitable for the user, a setting for external notification of incoming messages, and a setting for arbitrarily configuring multiple displays and displaying images in split mode.

[0044] [Structure 2]

[0045] A landscape image recording and reproducing system comprises: a landscape image recording and reproducing unit (10); a user data file (DATA-A) output from the landscape image recording and reproducing unit (10) and related to a user; a server (11) connected to the Internet; and an Internet data file (DATA-B) of given data obtained from the Internet, wherein the server has an inference function, which outputs recommended landscape image data (DATA-C) preferred by the user according to a given algorithm based on or based on the input of data from the user data file (DATA-A) and the Internet data file (DATA-B). The server (11) has a computing device. The server (11) may be an external server, or part or all of the server's functions may be built into the landscape image recording and reproducing unit (10). The server may also perform inference, deduction or prediction based on a given algorithm. The server may also utilize AI (artificial intelligence).

[0046] [Structure 3]

[0047] A scenic image recording and reproducing device (10) is a scenic image recording and reproducing device (10) having a display (1), the scenic image recording and reproducing device (10) having: a user data file (DATA-A) related to a user; a server (11) connected to the Internet; and an Internet data file (DATA-b) of given data obtained from the Internet, having an inference function, the inference function outputting recommended scenic image data (DATA-C) preferred by the user according to a given algorithm based on or on the input of data from the user data file (DATA-A) and the Internet data file (DATA-B). By integrating the server (11) into the scenic image recording and reproducing device (10), the inference function can be improved without being affected by external communication conditions, for example.

[0048] [Structure 4]

[0049] A method for recording and reproducing a landscape image comprises the following steps: creating a user data file (DATA-A) related to a user from a landscape image recording and reproducing unit (10); creating an Internet data file (DATA-B) of given data obtained from the Internet; and outputting recommended landscape image data (DATA-c) preferred by the user according to a given algorithm based on or in accordance with the input of data from the user data file (DATA-A) and the Internet data file (DATA-B).

[0050] [Structure 5]

[0051] The user data file (DATA-A) includes at least one of the following information: information of the user, scenery viewing history, operation history; user status obtained by camera, microphone, sensor, etc.; status of the setting environment obtained by camera, microphone, sensor, radio wave conditions, etc.; and location information of the setting place of the scenery image recording and reproducing unit. In some embodiments, the Internet data file (DATA-B) may also include at least one of the date and time when the recommended scenery is output from the server, season, time information, weather, map information, calendar information, SNS information of the user, news, and scenery information.

[0052] [Structure 6]

[0053] An image reproduction device (10) comprises: a display (1) for displaying images; a memory (4) for storing images; an Internet interface (7) for reading image data stored in the Internet (12); a decoding chip (14); and a video memory (23) for reading image data stored in the memory (4) and / or the Internet (12); decoding processing (27) is performed on the read image data by the decoding chip according to the format of the image data; a frame image obtained by the decoding processing is stored in the video memory (28) for each image frame (F1, F2, . . .); and the frame image stored in the video memory (28) is displayed on the display (1) in the order of the image frames (F1, F2, . . .).

[0054] [Structure 7]

[0055] The image reproduction device (10) according to structure 6 is characterized in that it also has a random access memory (5) for storing the frame image after the decoding process, performs a first image processing (29) on the frame image stored in the random access memory (5), stores the frame image after the first image processing in the video memory (28), performs a second image processing (30) on the frame image stored in the video memory (28), and displays the frame image after the second image processing on the display (1).

[0056] [Structure 8]

[0057] A method for reproducing an image comprises the following steps: reading in image data stored in a memory of an image reproducing device (10) and / or image data stored in the Internet (12); decoding (27) the read image data (26) into a frame image according to the format of the image data; storing the decoded frame image in each image frame (F1, F2, . . .); and displaying the frame image on the image reproducing device in the order of the image frames (F1, F2, . . .).

[0058] [Structure 9]

[0059] An image reproduction device and a reproduction method, comprising: a unit for adjusting the image quality in real time when reproducing a landscape image based on the image quality or characteristics of the landscape image, the image quality or error of a display, the surrounding environment information obtained from a sensor, etc., the information set by the user, the information of the preferred image quality estimated based on past statistical information, etc. In some embodiments, a GPU or a dedicated chip can be used in addition to a CPU so as not to reduce the frame rate. Thus, for example, the large load of real-time image processing of the image can be reduced.

[0060] [Structure 10]

[0061] A display method comprises the following steps: displaying a desired scene on a display; setting spatial information of the scene; receiving a notification from an external source; mapping the notification from an external source into an object that can be inserted into an image space; and dynamically inserting the object into the image. The object can be a dynamic image or a static image. In some embodiments, the dynamic insertion of the object can be at least one of fade-in, slide-in, and slow-in.

[0062] [Structure 11]

[0063] A display device comprises: a display (1) for displaying a desired scene; a unit for setting spatial information of the scene; a unit for receiving notifications from the outside; a unit for drawing the notifications from the outside into an object that can be inserted into an image space; and a unit for dynamically inserting the object into the image.

[0064] [Structure 12]

[0065] A display method is a method for displaying an image in an image display device (40) having a display (1), a camera (41) and a computer graphics generation function, characterized in that it comprises the following steps: identifying the position of a viewer in which the camera (41) is located in front of the display (1); and moving the display image of the display (1) in conjunction with the movement of a given part of the viewer.

[0066] [Structure 13]

[0067] The display method according to structure 11 is characterized in that it includes the following steps: tracking the movement of a given part of the viewer, setting a virtual camera (42) of the camera (41) on a computer graphic to match the position of the viewer; and displaying a scene image that can be seen from the virtual camera (42) on the display (1). The given part of the viewer can be any one or a combination of the face, eyes and other body parts.

[0068] [Structure 14]

[0069] An image display device comprises: a display (1); a camera (41) for identifying the position of a viewer in front of the display (1); and a computer graphics generation unit for generating an image of the display (1) and moving the display image of the display (1) in conjunction with the movement of a given part of the viewer.

[0070] [Structure 15]

[0071] A display method comprises the following steps: detecting positions of a plurality of displays arranged arbitrarily from an image obtained from the outside; cutting out an image using the detected positions of the displays; and synchronously displaying the cut-out images on the displays at the respectively detected positions.

[0072] [Structure 16]

[0073] The image display method according to structure 15 has the following steps: setting one of the multiple displays as a dominant display and setting the other displays as following displays; forwarding the reproduction position information of the dominant display to the following display; and when the following display receives the reproduction position information from the dominant display, cutting out the same image for display in the multiple displays respectively.

[0074] [Structure 17]

[0075] An image display system comprises: a plurality of displays arranged arbitrarily; an electronic device for acquiring an image including the plurality of displays; a unit for detecting the positions of the plurality of displays in the acquired image; and a unit for cropping an image from the detected positions of the plurality of displays and displaying it on the displays.

[0076] In the following embodiments, a landscape image recording and reproducing device is described as an example of the present disclosure. However, the present disclosure is not limited to the landscape image recording and reproducing device, and can also be applied to an image display device.

[0077] For example, Figure 1 1 is a simplified block diagram showing a landscape image recording and reproducing system according to an embodiment of the present disclosure. The landscape image recording and reproducing system 100 includes an image recording and reproducing apparatus 10, the Internet 12 connected to a server 11, and an external device 13, which are wirelessly connected to each other.

[0078] The external device 13 can be, for example, a smart device that is connected to and can cooperate with a wireless communication system such as Wi-Fi BT, and can be a smart phone, a smart speaker, a smart display, or a device that can be connected to the Internet such as a PC. The smart phone can also be configured to perform basic operations such as landscape operation, and the smart speaker can emit sound from a preferred position. In addition to wireless communication devices, control devices such as infrared remote controllers can also be used.

[0079] The image recording and reproducing device 10 is configured to include: a display 1; a GPU (graphics processing unit) 2 for displaying images on the display 1; a CPU 3; a ROM 4; a RAM 5; a sensor interface 6 (realized by USB, SPI, I2C, UART, CAN, etc.) for installing various sensors such as a camera 8 and a microphone 9 (other sensors are not shown); and a network interface 7 (realized by Wi-Fi (registered trademark), Bluetooth (registered trademark), LTE (registered trademark), wired LAN, etc., and the display of registered trademarks is omitted for the same terms in this specification) for communicating with an external device 13, the Internet 12, etc. The reproduced landscape image exists in the ROM 4 in the image recording and reproducing device 10, or in a storage device (not shown) in the server 11 on the Internet 12. When reproducing images on the Internet 12, streaming reproduction can be performed while storing the cache in the ROM 4, and cache reproduction can be performed when continuously reproducing. As a result, for example, Internet communication is not required.

[0080] In this embodiment, the computing device in the server 11 on the Internet 12 determines the recommended scenery and notifies the image display device 10. Thus, the recommended scenery can be automatically reproduced. In some embodiments, the server 11 may also be provided outside the image recording and reproducing device 10. In some embodiments, part or all of the server functions including the inference function may also be provided outside the image recording and reproducing device 10 or inside the image recording and reproducing device 10.

[0081] Reference Figure 2 This section explains how to determine the recommended scenery. Figure 2 In the hardware structure and Figure 1 The image display device 10 has the same Figure 1 The image display device 10 includes a display 1 and connected components (except GPU 2).

[0082] In order to determine the recommended scenery, as shown by arrow (1), data 21 (DATA-A) required for scenery selection is sent from the image recording and reproducing device 10 to the server 11 via Internet communication. The server 11 accesses the Internet 12 as shown by arrow (2) to collect necessary data 22 (DATA-B) other than the data 21 (DATA-A), and processes the information data 21 and 22 by a given recommended scenery selection algorithm or AI, thereby determining data 23 (DATA-C) as a candidate for recommended scenery, and sends it to the image recording and reproducing device 10 along arrow (3).

[0083] Data 21 (DATA-A) transmitted from video recording and reproducing device 10 to server 11 includes the following information DA1 to DA5.

[0084] DA1: Exploiting User Information

[0085] The term "information of the user using" in the information DA1 used in this specification refers to the information of the user currently viewing the display 1 of the video recording and reproducing device 10. The video recording and reproducing device 10 may have a user registration function, and the user may be managed by the video recording and reproducing device 10 and the server 11. It is possible to understand which user is using the video recording and reproducing device.

[0086] The image recording and reproducing device 10 used by a single user sends the user information being logged in to the server 11. The image recording and reproducing device 10 used by multiple users may send the user information being logged in, or may analyze the image of the camera image of the camera 8 of the image recording and reproducing device 10 and send the information of the user located before the image recording and reproducing device 10 to the server 11.

[0087] DA2: Scenery viewing history, operation history

[0088] The term "scenery browsing history, operation history" used in this specification refers to information on the history of what scenery the user has viewed, what operations the user has performed, etc. This history information is sent to the server 11. The history information is used as data for inferring the user's interests and hobbies in the server 11 and outputting recommended scenery candidates. The history information can be sent to the server 11 each time a scenery is browsed or operated, or it can be sent collectively when a scenery is selected.

[0089] For example, the history information can be used to increase the proportion of sea or summer scenery when inferring the recommended scenery candidates for users who often watch the scenery of the sea. For example, the history information can be used to reduce the proportion of scenery including animals when inferring the recommended scenery candidates for users who often perform operations to change the scenery immediately in the case of animal scenery.

[0090] DA3: User status obtained by camera, microphone or sensor

[0091] The term information DA3 "user's status obtained by the camera, microphone or sensor" used in this specification is information that infers the user's status based on data from the camera 8, microphone 9, and sensor (not shown) of the image recording and reproducing device 10. This can also be used to output a scenery candidate that matches the user's status when inferring a candidate for a recommended scenery. For example, in a case where it can be inferred that the user is concentrating while looking down at a desk, a "calm scenery that allows concentration" can also be output as a scenery candidate. For example, in a case where the user can infer that many people are having dinner or at a party, a "lively and bright scenery" can also be output as a scenery candidate. For example, in a case where the user can infer that they are in a jazz bar or drinking quietly, a "night scene" can also be output as a scenery candidate. For example, in a case where it can be inferred that there is no one, a "quiet scenery" can be output as a scenery candidate, and the image recording and reproducing device 10 can be instructed to sleep as needed.

[0092] DA4: Status of the setting environment obtained using cameras, microphones, sensors, or radio wave conditions

[0093] The information DA4 "the state of the installation environment obtained by using a camera, microphone, sensor or radio wave condition" used in this specification refers to information related to the environment in which the image recording and reproducing device 10 is located, which is obtained and estimated by using the camera 8, microphone 9, other sensors (not shown), radio wave conditions, etc. of the image recording and reproducing device 10. This information is used to output a landscape candidate that matches the user's state when inferring a candidate for recommended landscape in the server 11.

[0094] Information about the environment obtained by using the radio wave conditions and inferred based on the conditions is, for example, the result information obtained by using the following method when there are many people who own wireless devices such as smartphones and PCs, wherein the method uses the phenomenon of deterioration of radio wave reception sensitivity in the environment and disconnection from the router to measure the amount of activity. In the case of an environment such as an office area, the information of "radio wave conditions" based on the SSID (connection name) and number of routers can be used in operations such as inferring the environment where the router is set up based on the characteristics such as the large number of routers (SSID). The wireless reception sensitivity, the status of the SSID of the router, etc. can also be saved regularly. The saved information can be used to infer the amount of activity or environment around the router. This information can be used to infer the scenery. For example, in the case of inferring that there are many human activities, lively scenery such as streets or amusement parks can also be displayed. For example, in the case of inferring that it is an office area, a quiet forest can be displayed.

[0095] For a quiet and calm environment, "quiet and gentle scenery" can be output, and for a busy environment, "scenery that attracts you at first sight" can be output. When an office environment is estimated based on the camera 8, radio wave conditions, etc., "scenery that makes it easy to concentrate on work" can be output. When a home environment is estimated based on the camera 8, radio wave conditions, etc., "scenery that makes people relax" can be output as a scenery candidate. When the environment is estimated to be turned off, an instruction to instruct the image recording and reproducing device 10 to sleep can also be output.

[0096] The sensor for estimating the environment of the image recording and reproducing device 10 may also be a thermometer, a barometer, an altimeter, a geomagnetic sensor, an illuminance meter, a LIDAR, an odor sensor, an atmospheric sensor, etc., which are provided inside and / or outside the image recording and reproducing device 10. Radio waves may also be used to grasp the positional relationship with a router through Wi-Fi positioning, or to transmit environmental information to the image recording and reproducing device 10 through a beacon (such as Bluetooth) or an RFID.

[0097] DA5: Set the location information of the place

[0098] The term "location information of the installation location" of the information DA5 used in this specification refers to the location information of the installation location of the image recording and reproducing device 10 obtained by user settings, GPS sensors, base station information of the SIM based on the network I / F, etc. This information may also be information obtained by inference on the placement of the image recording and reproducing device 10. This information DA5 is used to output scenery candidates that match the user's status when the server 11 infers the recommended scenery candidates. For example, in the server 11, when the local sun position is known based on the latitude and longitude of the installation location of the image recording and reproducing device 10, scenery candidates that match the sun position can also be output (sunrise scenery at sunrise, daytime scenery during the day, evening scenery at sunset, after sunset or night scenery, etc.). When the season of the installation location is known based on the latitude and longitude and the date and time, scenery candidates that match the known season can also be output (summer scenery in summer, etc.). When the country or place of the setting location is confirmed based on the latitude and longitude, scenery candidates matching the confirmation can also be output (using calendar information, if the device 10 is set in Japan, scenery related to the day of an event unique to Japan (such as the Peach Blossom Festival) is output).

[0099] The data 22 (DATA-B) collected by the server 11 from the Internet 12 includes the following information DB1 to DB9.

[0100] DB1: Date Time

[0101] The term "date and time" in the information DB1 used in this specification refers to information on the date and time of the recommended scenery requested by the operation of the video recording and reproducing device 10. This information is combined with the location information of the installation place of the aforementioned DA5 and other information to be used when the server 11 outputs scenery candidates that match the date and time, season, etc., of the local installation location of the video recording and reproducing device 10.

[0102] DB2: Seasons

[0103] The term "season" in information DB2 used in this specification refers to information related to the season of each country or place. As described in the aforementioned DA5 "location information of the installation location" and DB1 "date and time", this information is used when the server 11 outputs scenery candidates that match the season of the installation location of the image recording and reproducing device 10 by combining it with the season of the installation location of the image recording and reproducing device 10. The "season" of the installation location of the image recording and reproducing device 10 is estimated based on the DA5 "location information of the installation location" and DB1 "date and time", and is used when outputting scenery candidates related to the "season" of the installation location of the image recording and reproducing device 10 based on the estimated information, that is, information related to the season recorded on the Internet including the server. For example, if it is set in Japan, on January 1, the scenery related to the first month of the year can be displayed in the image recording and reproducing device 10.

[0104] DB3: time information (sunrise, sunset, moon age, etc.)

[0105] The term "time information (sunrise, sunset, moon age, etc.)" in the information DB3 used in this specification refers to information related to the time of each country or place, such as the time of sunrise, sunset, moon age, etc. As described in the aforementioned DA5 "location information of the setting place", the recommended scenery candidate is used when the server 11 outputs the scenery matching the time. For example, regarding the scenery matching the time, when the request time of the recommended scenery candidate is sunrise, the sunrise scenery is output, and after sunset, the night scenery is output, etc.

[0106] DB4: Weather

[0107] The term "weather" in the information DB4 used in this specification refers to weather information at the location where the image recording and reproducing device 10 is installed. For example, when a command is operated to display a recommended scenery as an image of the image recording and reproducing device 10 when it is raining, a rainy scenery is output, and the server 11 outputs a candidate scenery that adds rain or raindrops to the scenery image with CG to add a performance such as looking out of an actual window, and the weather information is used at this time.

[0108] DB5: Map Information

[0109] The term "map information" of information DB5 used in this specification refers to map information that can be obtained from Google Map or the like. This information, as described in the aforementioned DA5 "location information of the installation location", can determine the installation location of the device 10 on the map by combining the location information of the image recording and reproducing device 10 with information DB5, and collect information on the country or region and map or geographical features. Based on the collected information, the server 11 outputs landscape candidates suitable for the country or region where the device 10 is installed. For example, in the case where a color or object specific to the culture of the country is not preferred, landscape candidates that do not include unpreferred landscapes may be output.

[0110] DB6: Calendar information

[0111] The term "calendar information" of DB6 used in this specification refers to calendar information for each country or region that can be obtained through Google Calendar, etc. The server 11 grasps national or regional events (Christmas, Japan's Peach Blossom Festival, etc.), and outputs landscape candidates that take the events into consideration when output of recommended landscapes is requested.

[0112] DR7: SNS

[0113] The term "SNS" in the information DB 7 used in this specification refers to information using SNS of users, etc. The server 11 estimates the user's hobbies or global trends by analyzing the information DB 7, and outputs landscape candidates taking the estimated information into consideration.

[0114] DB8: News

[0115] The term "news" in the information DB8 used in this specification refers to information related to news of the place where the video recording and reproducing device 10 is installed or world news. Based on the information DB8, the server 11 outputs scenery candidates that match the time and season when the device 10 is operating. For example, in the Olympics, many scenery candidates related to sports or venues are output.

[0116] DB9: Scenery Information

[0117] The term "scenery information" in the information DB 9 used in this specification refers to information related to the shooting location or shooting object of each scenery image, etc. The server 11 grasps the details or background of the scenery image and uses this information as reference information when estimating recommended scenery.

[0118] The information DA1 to DA5 and DB1 to DB9 that does not change may be stored in the server 11. This eliminates the need for retransmission or collection of information (data), for example, and can reduce processing time and load when selecting (inferring) recommended scenery.

[0119] exist Figure 2 In the embodiment of the invention, the data 23 (DATA-C) indicated by the arrow (3) indicating the flow of data from the server 11 to the video recording and reproducing device 10 is data of "recommended scenery candidates" and is outputted by the server 11 based on a given recommended scenery selection algorithm. According to an embodiment of the given recommended scenery selection algorithm, in addition to the embodiments described in relation to the data 21 (DATA-A) and the data 22, the following two methods can also be used.

[0120] In the first method, the server 11 regards "scenery with many views" and "scenery with many collections set in the collection setting function" as popular scenery, and preferentially outputs the popular scenery as scenery candidates.

[0121] In the second method, the server 11 semi-automatically sets a recommendation function, so that the user notifies the keyword of the scenery that the user wants to be recommended through the image recording and reproducing device 10, and the notified keyword is preferentially output as a recommended scenery candidate. For example, the keyword can be "spring" or "summer" if the user wants to see scenery in a specific season, "sea" or "mountain" if the user wants to shoot the subject, and "cat" or "tower" if the user wants to enter the scenery. Multiple keywords can also be specified. Their relationship can also be specified by "and" or "or". The means of specifying the keyword can also be "selecting from the candidates in the image recording and reproducing device", "inputting a character string into the image recording and reproducing device", "sound input using a microphone", "reading a QR code (registered trademark) with the keyword set by the camera", etc. In addition, by "reading similar scenery with a camera", "setting the URL of the similar scenery in the image recording and reproducing device (by manual input, sound input using a microphone, QR code (registered trademark) using a camera, etc.)", the scenery similar to the set similar scenery can be output as a recommended scenery candidate.

[0122] Including the first and second methods, a plurality of algorithms described in the present disclosure may be combined, and may be combined with adding priorities.

[0123] The implementation form of the recommended scenery selection algorithm in the server 11 as the recommended scenery candidate outputting data 23 (DATA-C) can be programmed as a fixed algorithm or AI technology can be used. In particular, deep learning is cited as an embodiment in which data 21 and data 22 are used as input data and data 23 is used as output. By understanding the user's behavior when the recommended scenery is displayed on the device 10 (replaying for a long time, setting it as a favorite, changing it to another scenery immediately, etc.), it is possible to infer whether the displayed scenery is liked by the user. Therefore, by continuously learning the inferred results as correct data, it is possible to improve the model and improve the quality of the recommended scenery function.

[0124] The function of recommending recommended scenery can be implemented through the user interface (UI) at the following timing.

[0125] Timing 1: Spontaneous landscape switching by the user

[0126] It is the timing at which the user issues a scenery switching instruction using interfaces such as operation buttons (wired or wireless, not shown) respectively provided on the image recording and reproducing device 10, a remote control of the image recording and reproducing device 10 (one of the external devices 13), gesture input using the camera 8, and voice input using the microphone 9.

[0127] Timer 2: Automatically switch the scene based on the passage of time

[0128] It is a timing for switching scenery regularly and / or at a specified time. It can also be specified with power off and power on. For example, the scenery is switched by turning on the power at 8 am, and the scenery is automatically switched without user operation by turning off the power at 8 pm. The user can set the switching interval or time, and the function to be implemented with the switching of power off / power on, etc. in the image recording and reproducing device 10 or the server 11. The setting means is the same as the above-mentioned timing 1 "switching the scenery spontaneously by the user".

[0129] Next, the screen display performed by the display 1 of the video recording and reproducing device 10 for processing and display will be described with reference to FIG. 3 .

[0130] "Selection of scenery" and "reproduction of scenery" as recommended scenery are separate and different processes. The process using data files 21 and 22 is "selection of scenery", and the screen is not displayed during the selection process. When the selection process is completed and the next scenery is determined in the server 11, the process is transferred to the "reproduction of scenery" for the next scenery, and the scenery is reproduced on the screen in sequence as shown in display steps a1 to a4 of Figure 3(a). As described above, the reproduced scenery image is stored in the ROM4 in the image recording and reproducing device 10 and / or in the storage in the server 11 on the Internet 12. When reproducing the image on the Internet 12, streaming reproduction is performed while storing the cache in the ROM4. Cache reproduction is performed during continuous reproduction. Therefore, the "reproduction of scenery" process does not require Internet communication.

[0131] The scenery determined by "scenery selection" is reproduced in the form of a unique symbol following the "scenery reproduction" process. As the unique symbol, for example, a numeric code, a character code, a URI, etc. are used without limitation. By using the above symbol, the scenery data stored in the server 11 or the scenery data in the cache can be identified and reproduced.

[0132] Specifically, when the next scene needs to be switched in the display step a1 of FIG. 3(a), the image recording and reproducing device 10 performs the "scenery selection" process without changing the screen. When the next scene to be reproduced is determined, the image recording and reproducing device 10 performs the "scenery reproduction" process using the symbol representing the determined scene. At the time point when this process is completed, the process moves to the display step a2. By repeating this process flow as the display steps a2, a3, and a4, the display of the display steps a1 to a4 is performed.

[0133] In the display step b1 of FIG. 3( b ), after the “scenery selection” process, a list of candidate scenery is displayed. The list of scenery is created by the candidates for “scenery selection” and their thumbnails (reduced images). In some embodiments, the user can select a specific scenery from the list to display. In some embodiments, the user can use the list as a scenery reproduction list so that it is displayed and reproduced in sequence from display steps b2 to b5. If there is a scenery that is not the user's preference, an instruction for the scenery that is not the user's preference may be sent to the server 11 via the image recording and reproduction device 10, the scenery that is not the user's preference is removed, and then the recommended scenery candidates are output to the server 11 again.

[0134] In the case of the second method described above, "semi-automatically recommending by notifying the user of keywords of scenery to be recommended", recommended scenery is presented based on the notified keywords as shown in FIG. 3(c).

[0135] In FIG3(c), supplementary information is added to the "list of candidate scenery" in FIG3(b). For example, in the display step c1, a title such as "Summer" is added. Thus, for example, the user can easily understand the basis for selecting the candidate scenery.

[0136] Supplementary information becomes Figure 2 When the server 11 returns the data 23 of the scenery candidates, the server 11 includes the information of the data 21 or the data 22 for selecting the candidate as metadata (data additional to the data) in the data 23. The image recording and reproducing device 10 can display an image based on the supplementary information as in the display step c1 of FIG. 3(c) using the metadata.

[0137] Thus, FIG. 3(c) shows an example in which the keyword "summer" or "summer" is specified. In the display step c1, a list of recommended scenery related to summer is displayed. The user can select a scenery to be reproduced from the list of scenery, or can sequentially reproduce the list of scenery in the display step as a reproduction list as in the display steps c2 to c5.

[0138] Figure 4 FIG. 1 is a simplified block diagram showing the structure of an image reproduction device 10 according to another embodiment of the present disclosure. Figure 1 The same configuration as the image recording and reproducing device 10 of the scenery image recording and reproducing system 100 is used, and the same reference numerals are given to the same configuration elements.

[0139] The image reproduction device 10 includes: a display 1; a GPU for displaying an image on the display 1; a CPU; a ROM; a RAM; a sensor interface (sensor I / F: composed of USB, SPI, I2C, UART, CAN, etc.) 6 for installing various sensors; and a CPU for communicating with an external device (equivalent to a CPU). Figure 1 The display 1 can be mounted on a wall and used as a virtual window. The landscape image to be reproduced is stored in the memory (ROM) 4 in the image reproduction device 10 and / or the storage device (equivalent to the storage device) on the Internet 12. Figure 1 In order to reproduce the video on the Internet 12, the streaming reproduction is performed while storing the cache in the memory 4. Thus, the cache reproduction is performed during the continuous reproduction, and the Internet communication is not required.

[0140] A dedicated chip 14 for realizing the present embodiment is built into the video playback device 10. The dedicated chip 14 includes the following necessary chips for the functions to be realized.

[0141] Image decoder chip: In order to reduce the load of real-time image processing, a chip dedicated to image decoding processing is used. This chip can reduce the load of CPU or GPU decoding processing. Suitable for H.265 decoder chips, etc.

[0142] Image processing chip: A chip that is specifically responsible for image processing of image frames in order to reduce the load of real-time image processing. This chip can reduce the image processing load of the CPU or GPU. Depending on the needs, image processing dedicated hardware accelerators, programmable accelerators, DSPs, image processing dedicated CPUs, etc. are used.

[0143] AI chip: A dedicated chip used when AI processing is performed during image processing in order to reduce the load of real-time image processing. AI chips are mainly used for inference processing. When learning is also performed in the device 10, a learning chip is installed.

[0144] Figure 5 The following is a schematic diagram showing a general processing flow in the image reproduction device 10. The image data 26 obtained from the memory (ROM) 4 or the Internet 12 is subjected to a decoding process 27 corresponding to the format of the image data 26, and the image or sound of each frame F1, F2, F3, F4, ... is extracted. The decoding process 27 is performed by the CPU 3 or a dedicated chip (H.265 decoding chip, etc.) 14. The frame image is stored in the video memory 28 and displayed on the display 1. The process of displaying the image of the video memory 28 on the display 1 is basically performed by the graphics processing unit (GPU) 2. The frame display is repeatedly performed in accordance with the frame rate of the image (24fps, 29.97fps, 60fps, etc.) (for example, if it is 24fps, 24 frame images are displayed in 1 second), and the sound is reproduced in accordance with the repeated display, thereby reproducing the image and sound.

[0145] The video memory 23 generally uses the RAM 5 or the built-in memory of the GPU 2. In addition, if preferred, the video memory built into the dedicated chip 14 may also be used.

[0146] Figure 6 Show Figure 5 The conventional processing flow in the image reproduction device 10 is a modified processing flow chart in which two image processing 29 and 30 are added. The frame image resulting from the decoding 27 is not as Figure 5The image is directly sent to the video memory 28 as in the flow, first stored in the RAM 5, and the first image processing (Pro1) 29 is performed. This image processing is performed by the dedicated chip (image processing chip, AI chip) 14 when it is desired to reduce the CPU 3 or CPU load. The image after image processing is sent to the video memory 28, and the second image processing (Pro2) 30 is performed in the GPU 2 before the processed image is displayed on the display 1.

[0147] exist Figure 6 As an example, in the first image processing 29, the frame is binarized and fast-moving objects are deleted, such as the deletion of birds in frames FR1, FR2, etc., and in the second image processing 30, a scene in the rain seen from a window is performed on the display 1 (raindrops hanging on the window are displayed). If preferred, the sound of the image is also processed or corrected in accordance with the image processing.

[0148] Regarding the first image processing 29, the CPU 3 or the dedicated chip 14 is used to perform image processing on the images of each frame FR1, FR2, etc. However, it is needless to say that general image processing such as filtering processing can also be stored and placed if a large-capacity RAM 5 is used. Therefore, image processing using the transition of frame images (using moving object detection, moving object removal, smoothing using multiple frames, etc.) can also be implemented.

[0149] Information of other components in the device 10 can also be effectively utilized by passing through the RAM 5. Therefore, for example, image processing using the brightness or color of the surroundings of the device installation part or image processing using information on the Internet can be performed using the information of the camera center.

[0150] Other chips can also be used for image processing. Therefore, for example, AI chips can be used for image processing such as object recognition, face recognition, and text recognition, and image processing that matches the object. For example, a person can be detected and blurred so that the person cannot be identified. For example, characters can be recognized and automatically translated.

[0151] Regarding the second image processing 25, the frame images f1, f2, etc., which are output from the first image processing 29, are processed before being displayed on the display 1. However, basically, filtering processing using information of the frame image monomer before being displayed is mainly performed. Since the processing uses the pipeline of the GPU 2, various processing can be implemented programmatically using shaders. Image processing of the frame image can also be performed in Figure 6 Add other objects to the frame images listed in Figure 6 (additional raindrops in the middle).

[0152] In the image processing of the first image processing 29 and the second image processing 30, any image processing can be performed. Specifically, image correction, image transformation, and image processing of the frame image can be performed; other images, two-dimensional computer graphics (2DCG), and rendered three-dimensional computer graphics (3DCG) can be synthesized into the frame image; visual effects (VFX) can be added to the frame image, etc. Pre-processing such as image feature extraction and image recognition for realizing these can be performed at the same time. In some embodiments, the first image processing 29 and the second image processing 30 can implement all image processing. However, in some embodiments, by using two image processings in parallel to perform image processing at the same time, a higher frame rate can be maintained. In some embodiments, by assigning processing to each processor suitable for performing image processing of the first image processing 29 and the second image processing 30, processing efficiency can be improved. For example, image correction that performs the same processing on all pixels on the frame is assigned as a processor to GPU2, thereby enabling processing to be performed at a higher speed than when assigned to CPU3. For example, if it is AI processing, when an AI chip is loaded on the dedicated chip 14, high-speed processing can be performed by assigning it as a processor to the AI ​​chip.

[0153] The image processing of this embodiment is not a simple image processing, and is characterized in that the object is a landscape image. Since "the landscape is reproduced as a beautiful view seen from the window or a view preferred by the user", the image processing is performed on the landscape image in real time using the implementation unit as in the above-mentioned embodiment. The more specific content of the image processing is as follows.

[0154] <General Tone Correction of Images>

[0155] Depending on the camera or shooting environment, the overall color tone of the landscape image may be unfavorable. For example, the tone curve may be poorly balanced and biased to one side, or the white balance may be inappropriate. In this image processing, the overall image can be corrected to a more preferred color tone by automatically performing level correction, contrast correction, or color correction.

[0156] Figure 7 (a) shows an example of color tone correction of a general landscape image on the display 1. The original landscape image is dark as a whole, and the color tone curve is also biased toward 0. By automatically correcting the color tone curve so that it is balanced, it can be converted to a more preferred color tone.

[0157] Since the user has a preference for color tone, the image reproduction device 10 is configured to be able to change the presence, strength, or characteristics of the automatic correction according to the user's setting. As described below, the change can be made step by step based on the user's knowledge of the image quality.

[0158] (1) Individually and finely set brightness, contrast, etc.

[0159] (2) Set by general tone level ("soft", "bright", etc.)

[0160] (3) Choose a landscape with your preferred color tone

[0161] In some embodiments, the user's preferred color tone may be estimated based on the scenery that the user has viewed in the past, and the estimated preferred color tone may be displayed or notified during setting. This may, for example, be a reference for helping the user select the preferred color tone.

[0162] <Color tone correction corresponding to landscape content>

[0163] (1) In the color tone correction corresponding to the scenery content, the color tone correction suitable for the scenery is performed based on the shooting object of the scenery, the shooting time, etc. For example, the following color tone correction is performed.

[0164] (2) If the scenery mainly reflects mountains and forests, it will be corrected by emitting more intense green.

[0165] (3) If it is a seascape, make corrections to make the blue of the sea and / or the white of the clouds stronger.

[0166] (4) If it is a sunset scene, the color level is not corrected strongly, and the scene is corrected to be a red sunset as seen by the naked eye.

[0167] (5) If the scenery mainly reflects red leaves, modify it to emit red or yellow more strongly

[0168] Figure 7 (b) shows an example. The original landscape image is identified as a landscape of the sea, and the color tone is automatically corrected to make the blue of the sea and the white of the clouds more vivid. In some methods, the entire frame image can also be corrected. In addition, in some methods, a mask can be applied to correct only a part of the image (for example, Figure 7 In (b) only the cloud part is corrected).

[0169] In order to identify the content of the scenery, it is possible to use metadata of the scenery image (storing information such as the shooting location, shooting time, and supplementary information related to the scenery) and / or the scenery content grasped by image recognition (based on the estimation of similar scenery content retrieved through the image, inference based on a scenery recognition model (deep learning model) that has learned to read the scenery image and its shooting location, shooting time, etc.). Similar to the above-mentioned "general color tone correction of the image", due to the user's preference, the presence or absence or intensity of this correction can also be changed by user settings.

[0170] <Image Processing in Accordance with the Weather or Environment of the Device Installation Location>

[0171] When displaying a landscape image such as "a beautiful view of the scenery from the window", the image may be corrected according to the weather at the location where the image reproduction device 10 is installed. Thus, for example, the device can be perceived more like a window, and an illusion can be created that the scenery location actually exists nearby.

[0172] For example, if the device 10 reproduces a sunny scene when the weather is rainy, it is difficult to perceive the device 10 as a window. Figure 8 By adding a performance like raindrops attached to the window as shown in (a), the device 10 can be perceived as a window.

[0173] Similarly, by applying corrections or performances that match the weather such as sunny or cloudy, the sense of the window can be enhanced. Image processing or correction can also be performed based on environmental information other than weather, such as "if the surroundings of the device are dark, reduce the contrast" or "if the country where the device is placed does not prefer strong red, reduce the level of red." In order to understand the weather, information collected from the Internet 12, information obtained from sensors that are preferentially or wirelessly connected to the device 10, etc. can also be used. Similar to the above-mentioned "general color tone correction as an image", due to user preferences, the presence or absence or intensity of image processing corresponding to the weather or environment can also be changed in the user settings.

[0174] <Image processing that allows you to see the scenery through a special window>

[0175] Even the same scenery may look different when viewed from a frosted glass or stained glass window. The glass of the window functions as a filter, and the scenery can be seen to look different. The image obtained by subjecting the scenery to image processing, i.e., grayscale, binarization, sepia, cartoon style, painting style, ink painting style, etc., can be seen as an image whose appearance has changed when viewed through a window. As a result, the impression of the scenery changes greatly, and different impressions can be enjoyed. The scenery image can also be processed according to the user, a given device, or other settings so that the scenery can be seen through a special window.

[0176] Figure 8(b) shows an example of applying a given filtering process to a landscape image. For the original landscape image P1, an image P2 of "the landscape seen through a frosted glass window", an image P3 of "the landscape seen through a stained glass window", and an image P4 of "the landscape seen through a sepia window" are displayed. By operating the image reproduction device 10, the user can specify a special window or its features (the fogging of the frosted glass, the size of the mesh of the stained glass, etc.) according to preference.

[0177] According to this embodiment, the image reproduction device may also have a virtual window type display for displaying an image by projecting an existing scenery on the window. In order to display images such as "scenery seen through a frosted glass window", "scenery seen through a stained glass window", "scenery seen through a sepia window", etc., a step of synthesizing a given preferred glass window image with a scenery image is provided.

[0178] As another embodiment of the present disclosure, Fig. 9 An example of adding spatial information to an image is shown in FIG. Fig. 9 (a) represents the original image displayed on the display 1 of the image reproduction device, Fig. 9 (b) represents an image with spatial information. Fig. 9 (c) indicates an image to which new information is added. The image reproduction device of this embodiment comprises: a display 1 for displaying a desired scene; a unit for setting spatial information of the scene; a unit for receiving notifications from the outside; a unit for drawing notifications from the outside into objects that can be inserted into the image space; and a unit for dynamically inserting objects into the image. Figures 1 to 8 The image reproduction device 10 used in Fig. 9 In (b), a three-dimensional model related to perspective, that is, spatial information, is pre-set for the landscape image.

[0179] exist Fig.10 In FIG. 1 , the following example is shown: when some event occurs outside the image reproduction device, such as an incoming email or a notification in an SNS, the event is transformed into an object that can be inserted into the image model and dynamically inserted into the image.

[0180] The conversion from the above-mentioned event to the object is performed using, for example, the following rules, but may be individually set by the user.

[0181] (a) Manual association (converting taxi notification to taxi object).

[0182] (b) Notifications of services whose service color is blue are automatically converted into blue animals, etc.

[0183] (c) For notifications that should remain (such as taxi arrival), objects that remain are assigned, and for notifications that can disappear (such as SNS notifications), objects that disappear (such as animals) are assigned.

[0184] Fig. 9 (c) illustrates an example of waiting for the arrival of a reserved taxi according to the above rule (a). Based on the above multiple rules being known to the user in advance, when an object that is not normally seen in the scenery appears in the scenery, the user can recognize it as a notification. Fig.10 In (a), the scenery 31 is displayed on the display 1. Fig. 9 (b) Similarly, spatial information is given to the image. If there is an email from outside the playback device, Fig.10 As shown in (b), the incoming email notification is drawn as an object in the image space that can be inserted into the spatial information. The dynamic image of the object 32 of the bird holding the letter is synthesized (superimposed) on the scenery 31. In this way, the incoming email is naturally notified without obstructing the scenery 31. The notification can include not only the superposition of the object but also a given incoming message sound. The incoming email can be an incoming message to the image reproduction device or to a portable device such as a smartphone. The object can also be configured to indicate that it is received by a portable device. The object 32 can be a dynamic image of a bird flapping its wings or a static image. The dynamic insertion of the object 32 can also be selectively set to at least one of fade-in, slide-in and slow-in. In Fig.10 In the example of , the provision of spatial information to the landscape information is not limited to three dimensions but also includes two dimensions.

[0185] According to this embodiment, by combining a dynamic or static object representing a notification with a landscape image, the user viewing the landscape image can be notified without losing comfort. This embodiment is not limited to a display method for landscape images, but is also applicable to a display method for images other than landscape images.

[0186] As another embodiment of the present disclosure, Fig.11 (a) shows an example in which a scene is displayed on the display 1 when a person M stands in front of the image display device 40, and Fig.11 (b) shows an example in which a scene is displayed on the display 1 when a person M stands obliquely in front of the image display device 40 . Fig.11 (c) means Fig.11 The image of the camera view of the person M shown in the camera sensor device 41 included in the image display device 40 in the state (a) is shown in FIG. Fig.11 (d) means Fig.11 The image reflected by the camera sensor device 41 in the state (b). Fig.12 (a) and Fig.12 (b) indicates that Fig.11 (a) and Fig.11 The landscape images (b) correspond to CG (computer graphics) images created by an image processing unit built into the image display device 40.

[0187] The image display device 40 includes a display 1, a camera 41 for recognizing the position of a viewer M located in front of the display 1, and a computer graphics generation unit for generating an image of the display 1, and is configured to perform real-time CG rendering (image output after program calculation in real time) of an image of a landscape or the like, or to display a real-time image of a live camera on the display 1. The device used in the above-mentioned embodiment can be used. The image display that moves the display image of the display can be performed in conjunction with the movement of a given part of the viewer M.

[0188] exist Fig.11 In (a), the camera sensor device 41 can be used to extract the eyes or the outline of the person M in front of the face for face tracking, and the camera sensor device 41 can also be equipped with a depth sensor. Fig.11 When the image moves to the right as in (b), the image displayed on the display 1 changes as if the scenery seen from a window in the real world changes. For example, an object (image P2) that is not visible from the front becomes visible to the person M. Since the viewing angle of the person M changes, the field of vision (perspective) in the image also changes.

[0189] By the movement of the person M, the image on the display 1 can be displayed as if it is seen on the spot through the window of the display 1 in the shape of a window. Fig.12 In the process, it is necessary to calculate the camera (or live camera) 42 for real-time rendering inside the CG, Fig.12 (a) The front position of Fig.12 Therefore, it is necessary to first analyze the change distance d and the change angle α of the tilt position of person M. Fig.11 The position of (a) is Fig.11 The movement of the position of (b) is used to estimate the movement distance x of the person and the distance z from the camera sensor device 41 to the person's face (or eyes). Fig.12 The images p1, p2, and camera 42 correspond to Fig.11 Images P1, P2, camera sensor device 41.

[0190] First, the camera sensor device 41 is equipped with a depth sensor. If the distance z can be measured, the depth sensor is used. If the depth sensor is not equipped, Fig.11In (c) and (d), the distance z is calculated by comparing the size m of the eye extracted by face tracking with the length L of half the length of the image captured by the camera sensor device 41. The size of the eye can also be defined as the length from the tip of the eye to the outer corner of the eye (the opening of the eye). At this time, since there is a difference in the distance calculated based on the lens field of view of the camera sensor device 41, the coefficient k of the camera sensor device 41 is calculated in advance so that the distance z calculated based on m and L is the same as the actually measured distance. It is possible to calculate from Fig.11 The moving distance l from the center line of the image captured by the camera 41 to the center of the two eyes is calculated by adding the coefficient k mentioned above. Next, the angle θ from the center of the image display device 40 to the center of the two eyes of the person M after the movement can be calculated based on tanθ=x / z to obtain θ=arctanx / z.

[0191] exist Fig.12 In (b), when the moving distance of the camera inside the CG is limited to D, the moving distance d of the camera 42 inside the CG is calculated by d=D / Lxl. The angle α can be calculated as α=jxθ by considering the coefficient j that matches the camera characteristics of the CG (or the on-site camera) for the angle θ.

[0192] Based on the angle α and the movement distance d calculated as described above, the visible range of the image displayed on the device 40 , that is, the perspective (field of view) can be changed in real time with respect to the movement of the person M. This makes it possible to achieve, for example, a natural experience.

[0193] The above-mentioned on-site camera shoots outdoor scenery from indoors through the window. Figure 1 As described in , the images captured by the camera are always uploaded to the server and displayed on the display. The live cameras are set up all over the world, and the coefficient j changes according to the camera's field of view and other characteristics. The live camera moves / rotates in conjunction with the display camera 41 to change the perspective.

[0194] Fig.12 (a) and (b) are CG images. In the CG image that can obtain an image as if the camera 42 is moving, the dotted lines extending radially from the virtual camera 42 are equivalent to the graphics captured by the camera, which becomes Fig.11 When generating CG, the virtual camera 42 is moved to perform the above calculations.

[0195] According to this embodiment, in CG (for example, Fig.12 The image (a) of the image display device 40 is outputted, or an image obtained from an externally provided on-site camera is outputted. In the display 1, when the person M is located at a given position, an image limited to a given frame is displayed (for example, Fig.11 As the person M moves, the image moves in tandem, and a hidden image P2 appears. This allows the user to experience, for example, a natural change in the visible range through the window, as if they were looking through it.

[0196] According to this embodiment, the image display function of the visible range of the scenery and the perspective change in conjunction with face tracking can be achieved by making the position and angle of cameras such as cameras performing real-time rendering in CG (computer graphics) and live cameras linked to the human line of sight in real time.

[0197] In this embodiment, the person M moves laterally in parallel with the display 1. The camera 42 may be moved in the vertical direction in coordination with the movement of the person M in the vertical direction of the display 1 in the near and far direction. When the person M approaches the display, the camera 42 also approaches the frame. It is possible to display a wide-angle field of view including not only the image p1 but also the image p2. If the person M moves away, it is possible to display an angled field of view centered on the image p1. By moving the camera 42 in conjunction with the movement of the person M in another direction, the field of view of the scenery seen through the frame F1 can be changed. The viewer M can use the display 1 to experience looking at the outside scenery as if through an actual window. It is also possible to Fig.11 The display of the display 1 of (a) is in a default state, showing a standby state when no one is there, or showing a termination state when no one leaves.

[0198] In this embodiment, the structure of the image display device that changes the visible range and perspective of the scene in conjunction with the image by tracking the movement of the viewer's face (facial), but it is not limited to this. The movement of other parts of the body (such as the head or the whole body, etc.) can also be tracked.

[0199] As other embodiments of the present disclosure, Fig.13 An example of trimming display using position detection of the plurality of displays 51 , 52 , and 53 and synchronized reproduction on the plurality of displays is shown. Fig.13 (a) shows a cropped display using position detection of multiple displays, and Fig.13 (b) shows an example of synchronous reproduction of multiple displays 51, 52, 53. The displays 51, 52, 53 may include Figure 1 3, and a video recording and reproducing device 10 having an image processing, storage and reproduction function such as computer graphics. Although three monitors are described as a plurality of monitors, the number of monitors is not limited to three, and may also be applied to two or four or more.

[0200] exist Fig.13(a) shows an example of determining the size or position of the cropped image by using the relative distance between the display devices. In the cropping display method using position detection of multiple displays, at least any one of the following methods can be used as a method of obtaining the cropping position.

[0201] Method A: Detect the positions of the displays 51, 52, 53 based on images obtained by an external electronic device equipped with a camera, and calculate Fig.13 The dotted line in (a) encloses the cropping position of the display.

[0202] Method B: Using markers as an aid to the above method A, the clipping position of the display surrounded by the dotted line is similarly calculated.

[0203] Method C: The position is detected based on the moving distance from the initial position of the display 51, 52, 53, and the cropping position is calculated.

[0204] like Fig.13 As shown in (b), when the cropped image is displayed on a plurality of displays 51, 52, 53, they need to be synchronized. Fig.16 In the display example, as described later, the reproduction position information of the leading display among the multiple displays can also be transferred to the other following displays via Bluetooth or other communications to perform synchronization. In some embodiments, multiple displays can also be synchronized with each other. In the data forwarding of the communication, the display synchronization can be performed by the deviation of the communication delay time (waiting time) generated only from the issuance of the forwarding request to the actual sending of the data.

[0205] About Usage Fig.13 (a) A method of calculating a cropping position from the positions of the plurality of displays 51, 52, 53 is described in detail with reference to the cropping display of the plurality of displays 51, 52, 53 by position detection.

[0206] The physical positions of the plurality of displays 51, 52, 53 installed on a wall or the like are measured by any one of the above methods A, B, and C, or by position detection using other methods. The sizes of the plurality of displays may be the same or different.

[0207] Alternatively, the display position may be detected based on the image obtained by the electronic device of method A, and the cropping position may be calculated. Fig.14As shown in (a), an image is acquired by a camera of a smartphone, tablet computer, PC, etc. An electronic device capable of processing is used to acquire images of three displays 51, 52, and 53 set on a wall, etc. Next, based on the acquired images, the physical positions of the three displays such as the relative distances between the displays in the image are detected using methods such as pattern recognition of the displays. The image cropping position is calculated using an electronic device based on the acquired position information. The position information for cropping the image is sent to each display.

[0208] Reference Fig.14 (b) As an auxiliary to the above-mentioned method A, a method B for calculating the relative distance between the displays 51, 52, and 53 using a marker is described. As an auxiliary to the display position detection method of method A, a dot or a characteristic symbol K can be set on a wall or displayed on the display and used as a marker. The electronic device uses the marker in the image obtained by the camera to detect the physical positions of multiple displays more accurately.

[0209] Reference Fig.15 (a) of the present invention, a method C for detecting a physical position based on a moving distance from an initial position is described. First, for example, Fig.15 As shown in (a), the three displays 51, 52, and 53 are set to a determined initial position state, for example, by arranging them horizontally at the center in the vertical direction at the same time, and each display stores its own initial relative position.

[0210] Next, if Fig.15 As shown in (b), each display 51, 52, 53 is moved from the initial position to the desired position. In this case, the displays 51, 52, 53 can also be equipped with acceleration sensors, for example, to calculate the moving distance by acceleration detection, and detect the moving distance from the initial state. The relative position of each display can be calculated based on the detected moving distance. Based on the stored initial position and the position after the movement, the cropping position of the image is calculated, and the cropped image can be displayed.

[0211] In this embodiment, when the image cropping position is calculated based on the position detection methods A, B, and C, the image size can be set arbitrarily. Fig.16 The relative position and size ratio of the frames F1, F2, and F3 surrounded by dotted lines in (a) of FIG. 1 is used to calculate the cropping position. By sharing the calculated cropping position on multiple displays, the cropping position can be displayed as follows: Fig.16 The cropped image is displayed as in (b).

[0212] Next, refer to Fig.16 (b) describes a method for synchronizing and reproducing multiple displays 51, 52, and 53. When the display medium is an image, each display needs to synchronize the image reproduction position with each other while displaying a cropped image. Therefore, for example, one display 52 among multiple (three) displays 51, 52, and 53 is set as a leader. The leading display 52 forwards its own reproduction position information to multiple other displays 51, 53 that become followers via communication such as Bluetooth. After receiving the reproduction position information from the leading display 52, the following displays 51, 53 make the image reproduction position consistent with the leading reproduction position while each display crops and displays the same image. The leading display 52 can reproduce an image that is synchronized only by a deviation in the communication delay with the following displays 51, 53 by periodically sending the reproduction position information. That is, the image reproduction position of the leading display can be shared with the following displays, and a synchronized image can be reproduced.

[0213] The image display system of this embodiment comprises: a plurality of displays arranged arbitrarily; an electronic device having a camera for acquiring an image including the plurality of displays; a position detection unit for detecting the positions of the plurality of displays in the acquired image; and a display unit for cropping the image from the detected positions of the plurality of displays and displaying it on the display. As in the above-mentioned embodiment, the position detection unit and the display unit can be configured to Figure 1 Figure 3 Figure 5 , Figure 6 It is achieved by using a device and computer graphics processing.

[0214] In this embodiment, multiple displays are arranged at the user's preferred positions, and images obtained by cutting out the image of the scenery with the frame of each display are displayed on each display. In this way, for example, the user can be provided with a natural experience of viewing the scenery by setting a virtual window on the wall. Fig.11 , Fig.12 By combining the embodiment of face tracking, an image showing a video hidden in a blind spot of the display (virtual window) is provided in conjunction with the viewer's movement in front of the display. By combining with other embodiments, for example, the image display system of this embodiment can provide the user with a more comfortable viewing experience. This embodiment is not limited to landscape images, but can also be applied to general images.

[0215] The embodiments of the present disclosure are illustrative in all aspects, and the scope of the present disclosure includes the meaning equivalent to the claims and all changes within the scope. For example, the landscape image recording and reproduction system is not limited to the image recording and reproduction system, including image reproduction systems, devices, and methods without recording functions, and the image display device includes image recording devices, systems, and methods.

[0216] Several embodiments and examples of the present disclosure have been described above, but these embodiments and examples are illustrative of the present disclosure. For example, the above-mentioned embodiments are described in detail in order to easily understand the present disclosure, and the size, structure, material, and circuit may be additionally changed as needed. In addition, embodiments in which one or more of the features of the present disclosure listed above are arbitrarily combined are also included in the scope of the present disclosure. The claims include a plurality of variations of the embodiments without departing from the technical idea of ​​the present disclosure. Therefore, the embodiments and examples disclosed in this specification are shown for illustration purposes and should not be considered to limit the scope of the present disclosure.

[0217] -Explanation of Reference Numerals-

[0218] 1...Display

[0219] 2...GPU

[0220] 3...CPU

[0221] 4...ROM

[0222] 5...RAM

[0223] 8...Camera

[0224] 9...Microphone, etc.

[0225] 6...Sensor interface

[0226] 7...Network interface

[0227] 10...Image recording and reproducing device

[0228] 11...Server

[0229] 12...Internet

[0230] 13...External devices

[0231] 14...Specialized chip

[0232] 21...DATA-A

[0233] 22...DATA-B

[0234] 23...DATA-C

[0235] 24, 25... Image Processing

[0236] 26...Image data

[0237] 27...Decoding Processing

[0238] 28...Video memory

[0239] 29...First Image Processing

[0240] 30...Second Image Processing

[0241] 31...Scenery

[0242] 32...Object

[0243] 40...Image display device

[0244] 41...Camera (sensor device)

[0245] 42...Camera, Live Camera, Virtual Camera

[0246] 51, 52, 53...Display

[0247] 100...Landscape Image Recording and Reproduction System

[0248] a2, a3, a4...show steps

[0249] b2, b3, b4, b5...show steps

[0250] c1, c2, c3, c4, c5...show steps

[0251] M... people

[0252] P1, P2, P3, P4...Image

[0253] p1, p2...like.

Claims

1. An image display device, characterized in that: have: External input interface; A memory for storing image data; A processing unit; Graphics processing unit; a display for displaying images; Setting mode suitable for users; A unit for setting spatial information of a desired scenery; A unit for receiving notifications from the outside; Drawing the notification from the outside into a unit of an object that can be inserted into the image space; and A unit that dynamically inserts objects into images.

2. An image display method, using the image display device according to claim 1, characterized in that: have: displaying a desired scenery on the display; Set the spatial information of the scenery; Receive notifications from external parties; Drawing the notification from the outside into an object that can be inserted into the image space; and Dynamically insert objects into images.

3. The image display method according to claim 2, wherein: The object is a dynamic image.

4. The image display method according to claim 2, wherein: The object is a static image.

5. The image display method according to claim 2, wherein: The dynamic insertion of the object is at least one of fade-in, slide-in and slow-in.

6. An image display method, characterized in that: have: displaying a desired scenery on a display; Set the spatial information of the scenery; Receive notifications from external parties; Drawing the notification from the outside into an object that can be inserted into the image space; and Dynamically insert objects into images.

7. An image display device, characterized in that: have: a display showing a desired scenery; A unit for setting the spatial information of the scenery; A unit for receiving notifications from the outside; Drawing the notification from the outside into a unit of an object that can be inserted into the image space; and A unit that dynamically inserts objects into images.

Images