Image display device, system and method

Abstract

[Problem] To provide: an image display device comprising a display that is disposed at a user's preferred position and that displays an image of pleasant quality and arrangement, the image suitable to the viewing position and preferences of the user; and a method using the same. [Solution] An image display device (10) comprises: an external input interface (7), a memory (4) for saving image data, a computation processing unit (3), a graphics processing unit (2), a display (1), and a setting mode that is suitable to the user.

Description

[Technical field]

[0001] The present disclosure relates to an image display device and method having image playback and arrangement functions that match user preferences. [Background technology]

[0002] Conventionally, as an image display device that plays a user's favorite image, a video recording and playback device has been proposed that allows a user to send a command from a home viewing device to watch a favorite movie via a network (see, for example, Patent Document 1).

[0003] Also proposed is a method including the steps of capturing an image of a consumer with a video camera, biometrically scanning the image to determine whether a biometric can be recognized from the scan, and if a biometric is recognized from the scan, the system accessing a database containing a biometric associated with at least one of browsing history, purchasing history, and preference data associated with the consumer, and displaying an advertising image on a display device that matches the data stored in the database (see, for example, Patent Document 2).

[0004] As a display device that utilizes tracking of viewer parts such as the face, a stereoscopic image display device has been proposed that displays on a display panel a viewing zone image indicating whether or not the viewer is located in the viewing zone, based on the distance from the display device to the viewer and the viewer's position detected by a tracking unit (see, for example, Patent Document 3).

[0005] As a method for controlling multiple displays to display an image divided across multiple displays, a method has been proposed in which a host controller receives from the first display a first display state indicating the state of display of a first portion of the image by the first display, and receives from the second display a second display state indicating the state of display of a second portion of the image by the second display (see, for example, Patent Document 4). [Prior art documents] [Patent documents]

[0006] Patent Document 1: JP 2019-92152 A Patent Document 2: Patent No. 5893737 Patent Document 3: Patent No. 5050120 Patent Document 4: JP 2013-546042 A Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide an image display device and method having a display that is placed at a position preferred by a user and displays images with comfortable image quality and arrangement that matches the user's viewing position and preferences. [Means for solving the problem]

[0008] The present disclosure provides, as one aspect, an image display device including an external input interface, a memory for storing image data, an arithmetic processing unit, a graphics processing unit, a display, and a setting mode suitable for a user.

[0009] In another aspect, an image display device and a display method using the device are provided, which include a user data file related to a user output from the image display device, a server connected to the Internet, and an Internet data file of specified data obtained from the Internet, the server having an estimation function for outputting recommended landscape image data preferred by the user based on a specified algorithm in response to or based on input of the user data file and data in the Internet data file.

[0010] In yet another aspect, there is provided an image display device having a display for displaying video, a memory for storing video, an Internet interface for reading video data stored on the Internet, a decode chip, and a video memory, reading out video data stored in the memory and / or on the Internet, decode processing the read-out video data using the decode chip in accordance with the format of the video data, storing frame images generated by decoding each video frame in the video memory, and displaying the frame images stored in the video memory on the display in sequence for each video frame, and a display method using the device.

[0011] In another aspect, a display device and a display method using the device are provided, the display device having a display for displaying a desired landscape, a unit for setting spatial information of the landscape, a unit for receiving notifications from outside, a unit for mapping the notifications from outside onto an object that can be inserted into a video space, and a unit for dynamically inserting an object into the video.

[0012] In another aspect, an image display device is provided which includes a camera that recognizes the position of the viewer in front of the display and a computer graphics generation unit that generates an image of the display, and which moves the displayed image of the display in conjunction with the movement of a specific part of the viewer, and a display method using the device.

[0013] In another aspect, there is provided an image display device and a display method using the device, the image display device including a plurality of arbitrarily arranged displays, an electronic device for acquiring an image including the plurality of displays, a unit for detecting the positions of the plurality of displays in the acquired image, and a unit for clipping an image from the detected positions of the plurality of displays and displaying the image on the display. The clipped images can be synchronously displayed on the displays at the detected positions, respectively. Effect of the Invention

[0014] According to the present disclosure, for example, a display can be placed in a position of the user's preference, such as on a wall, and images with comfortable image quality and arrangement that match the user's viewing position and preferences can be displayed on the display. [Brief description of the drawings]

[0015] [Figure 1] 1 is a simplified block diagram of a landscape video recording and playback system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of the operation of the scenery video recording and reproducing system of FIG. [Figure 3(a)] 1A to 1C are views of landscape images sequentially displayed on a video recording and reproducing device as an example of the present embodiment. [Figure 3(b)] 11A to 11C are views of landscape images sequentially displayed on a video recording and reproducing device as another example of the embodiment. [Figure 3(c)] 11A to 11C are views of landscape images sequentially displayed on a video recording and reproducing device as another example of the embodiment. [Figure 4] 1 is a simplified block diagram of a video playback device according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a video processing flow diagram of the video playback device of FIG. 4. [Figure 6] FIG. 5 is a modified image processing flow diagram of the image playback device of FIG. 4; [Figure 7] 5(a) is an example of color correction of a landscape image displayed on the display of the image playback device of FIG. 4, and (b) is an example of color correction of a landscape image displayed on the display in accordance with the content of the landscape. [Figure 8] (a) is an example of image correction according to the installation environment of the image playback device in FIG. 4, and (b) is an example of a landscape image subjected to a specified filter processing. [Figure 9] 1A and 1B are examples of adding spatial information to an image displayed on a display according to one embodiment of the present disclosure, where (a) is the original image, (b) is the image with spatial information added, and (c) is the image with new information added. [Figure 10] Figure 9 shows an example of an image of an incoming external notification on a display. (a) is the original image, and (b) is the image when an email is received. [Figure 11]1A and 1B are front views of a display on which an image is displayed depending on the position of a person in front of an image display device according to one embodiment of the present disclosure, and FIGS. 1C and 1D are camera view views of the person. [Figure 12] 12(a) and (b) are computer graphic diagrams showing the camera arrangement of FIG. [Figure 13] FIG. 1A is an explanatory diagram of a cut-out display using position detection on multiple displays as an embodiment of the present disclosure, and FIG. 1B is a diagram showing the image cut out in FIG. [Figure 14] As an example, (a) is an explanatory diagram of a method for detecting the relative distance between multiple displays, and (b) is an explanatory diagram of another method for detecting the relative distance between displays. [Figure 15] As an example, (a) is a diagram showing the initial arrangement state of multiple displays, and (b) is an explanatory diagram of detection of display movement. [Figure 16] As an example, (a) is an explanatory diagram of changing the magnification of a cropped image, and (b) is an explanatory diagram of a method for detecting the relative distance between multiple displays. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. The present disclosure includes, for example, the following configurations.

[0017] [Configuration 1] An image display device comprising an external input interface (7), a memory (4) for storing image data, an arithmetic processing unit (3), a graphics processing unit (2), a display (1), and a setting mode suitable for the user. This device may include a picture quality setting suitable for the user, an external notification / incoming call notification setting, and a setting for splitting and displaying images by arbitrarily arranging multiple displays.

[0018] [Configuration 2] A scenic image recording and reproducing system comprising a scenic image recording and reproducing unit (10), a user data file (DATA-A) related to a user output from the scenic image recording and reproducing unit (10), a server (11) connected to the Internet, and an Internet data file (DATA-B) of predetermined data obtained from the Internet, the server having an estimation function of outputting recommended scenic image data (DATA-C) preferred by the user based on a predetermined algorithm in response to or based on input of data in the user data file (DATA-A) and the Internet data file (DATA-B). The server (11) has a computing device. The server (11) may be an external server, and some or all of the server's functions may be built into the scenic image recording and reproducing unit (10). The server, etc. may estimate, infer, or predict based on a predetermined algorithm. The server may utilize AI (artificial intelligence).

[0019] [Configuration 3] A scenic image recording and reproducing device (10) having a display (1), the scenic image recording and reproducing device (10) is provided with a user data file (DATA-A) related to a user, a server (11) connected to the Internet, and an Internet data file (DATA-b) of predetermined data obtained from the Internet, and the scenic image recording and reproducing device (10) is provided with an estimation function of outputting recommended scenic image data (DATA-C) preferred by the user based on a predetermined algorithm in response to or based on input of data in the user data file (DATA-A) and the Internet data file (DATA-B). By incorporating the server (11) in the scenic image recording and reproducing device (10), for example, the estimation function can be improved without being affected by external communication conditions.

[0020] [Configuration 4] A scenic image recording and playback method comprising the steps of: creating a user data file (DATA-A) related to a user from a scenic image recording and playback unit (10); creating an Internet data file (DATA-B) of predetermined data obtained from the Internet; and outputting recommended scenic image data (DATA-c) preferred by the user based on a predetermined algorithm in response to or based on the input of data in the user data file (DATA-A) and the Internet data file (DATA-B).

[0021] [Configuration 5] The user data file (DATA-A) has at least one of the following information: information on the user, scenery viewing history, operation history; user status acquired by a camera, microphone, sensor, etc.; installation environment status acquired by using a camera, microphone, sensor, radio wave condition, etc.; and location information of the installation location of the scenery image recording and playback unit. In some embodiments, the internet data file (DATA-B) may have at least one of the following information when the recommended scenery is output from the server: date and time, season, time information, weather, map information, calendar information, SNS information of the user, news, and scenery information.

[0022] [Configuration 6] A video playback device (10) having a display (1) for displaying video, an Internet interface (7) for reading video data stored in a memory (4) for storing video and on the Internet (12), a decode chip (14) and a video memory (23), which reads out video data stored in the memory (4) and / or the Internet (12), decodes (27) the read video data using the decode chip in accordance with the format of the video data, stores in the video memory (28) frame images decoded into each video frame (F1, F2, ...), and displays the frame images stored in the video memory (28) on the display (1) in sequence for each of the video frames (F1, F2, ...).

[0023] [Configuration 7] A video playback device (10) according to configuration 6, further comprising a random access memory (5) for storing the decoded frame images, performing a first image processing (29) on the frame images stored in the random access memory (5), storing the frame images after the first image processing in the video memory (28), performing a second image processing (30) on the frame images stored in the video memory (28), and displaying the frame images after the second image processing on a display (1).

[0024] [Configuration 8] A video playback method comprising the steps of: reading video data stored in a memory of a video playback device (10) and / or video data stored on the Internet (12); decoding (27) the read video data (26) into frame images in accordance with a format of the video data; storing the decoded frame images in each video frame (F1, F2, ...); and displaying the frame images in the video frames (F1, F2, ...) in order on the video playback device.

[0025] [Configuration 9] A video playback device and playback method equipped with a unit that adjusts the image quality in real time when playing back a landscape video based on the image quality or characteristics of the landscape video, the image quality or error of the display, surrounding environment information acquired from a sensor or the like, information set by the user, information on the preferred image quality inferred from past statistical information, etc. In some embodiments, a GPU or a dedicated chip may be used in addition to a CPU so as not to reduce the frame rate. This can reduce the heavy load of real-time image processing of the video, for example.

[0026] [Configuration 10] A display method comprising the steps of displaying a desired scene on a display device, setting spatial information of the scene, receiving an external notification, mapping the external notification to an object insertable into a video space, and dynamically inserting an object into the video. The object may be a moving image or a still image. In some embodiments, the dynamic insertion of the object may be at least one of a fade-in, a slide-in, and a slow-in.

[0027] [Configuration 11] A display device having a display (1) for displaying a desired landscape, a unit for setting spatial information of the landscape, a unit for receiving notifications from an external source, a unit for mapping the notifications from the external source onto an object that can be inserted into an image space, and a unit for dynamically inserting the object into the image.

[0028] [Configuration 12] A method for displaying an image on an image display device (40) equipped with a display (1), a camera (41), and a computer graphics generation function, comprising the steps of: recognizing the position of a viewer in front of the display (1) by the camera (41); and moving the displayed image on the display (1) in conjunction with movement of a predetermined part of the viewer.

[0029] [Configuration 13] The display method according to configuration 11, comprising the steps of: tracking the movement of a predetermined part of the viewer, setting a virtual camera (42) of the camera (41) on computer graphics to match the position of the viewer, and displaying an image of the scenery seen from the virtual camera (42) on the display (1). The predetermined part of the viewer may be any one or a combination of the face, eyes, and other body parts.

[0030] [Configuration 14] The image display device comprises a display (1), a camera (41) that recognizes the position of a viewer in front of the display (1), and a computer graphics generation unit that generates an image of the display (1), and is configured to move the displayed image of the display (1) in conjunction with the movement of a predetermined part of the viewer.

[0031] [Configuration 15] The display method includes a step of detecting the positions of a plurality of arbitrarily arranged displays from an image acquired from outside, a step of cutting out an image using the detected display positions, and a step of synchronously displaying the cut-out images on the displays at the detected positions.

[0032] [Configuration 16] An image display method according to configuration 15, comprising the steps of setting one of the plurality of displays as a leader display and setting the other displays as follower displays, transferring playback position information of the leader display to the follower display, and when the follower display receives the playback position information from the leader display, cutting out and displaying the same image on each of the plurality of displays.

[0033] [Configuration 17] An image display system comprising: a plurality of arbitrarily arranged displays; an electronic device for acquiring an image including the plurality of displays; a unit for detecting the positions of the plurality of displays in the acquired image; and a unit for cropping an image from the detected positions of the plurality of displays and displaying the image on the display.

[0034] In the following embodiment, a landscape video recording and reproducing device is described as an example of the present disclosure. However, the present disclosure is not limited to the landscape video recording and reproducing device and may also be applied to an image display device.

[0035] 1 is a simplified block diagram of a landscape video recording and playback system according to an embodiment of the present disclosure. The landscape video recording and playback system 100 includes a video recording and playback device 10, an internet 12 connected to a server 11, and an external device 13, which are wirelessly connected to each other.

[0036] The external device 13 is, for example, a smart device that can connect to and cooperate with a wireless communication system such as Wi-Fi or BT, and may be a device that can connect to the Internet, such as a smartphone, a smart speaker, a smart display, or a PC. The smartphone may be configured to perform basic operations such as landscape operation, and the smart speaker may be configured to emit sound from a preferred position. In addition to wireless communication devices, a control device such as an infrared remote control may be used.

[0037] The video recording and reproducing device 10 is configured to include a display 1, a GPU (graphics processing unit) 2 for displaying video on the display 1, a CPU 3, a ROM 4, a RAM 5, a sensor interface 6 (realized by USB, SPI, I2C, UART, CAN, etc.) for attaching various sensors such as a camera 8 and a microphone 9 (other sensors are not shown), an external device 13, a network interface 7 (realized by Wi-Fi (registered trademark), Bluetooth (registered trademark), LTE (registered trademark), wired LAN, etc. In this specification, the display of registered trademarks for similar terms is omitted.) for communicating with the Internet 12, etc. The scenery video to be reproduced exists in a ROM 4 in the video recording and reproducing device 10, or in a storage (not shown) in a server 11 on the Internet 12. When reproducing a video on the Internet 12, streaming reproduction is performed while storing a cache in the ROM 4, and cache reproduction may be performed during continuous reproduction. This, for example, eliminates the need for Internet communication.

[0038] In this embodiment, a computing device in a server 11 on the Internet 12 determines a recommended scene and notifies the video display device 10 of the determined scene. This allows the recommended scene to be automatically played back. In some embodiments, the server 11 may be provided outside the video recording / playback device 10. In some embodiments, some or all of the server functions including the inference function may be provided inside the video recording / playback device 10, rather than outside the video recording / playback device 10.

[0039] A method for determining a recommended scenery will be described with reference to Fig. 2. In Fig. 2, the hardware configuration is the same as that of the scenery video recording and playback system 100 in Fig. 1. The video display device 10 has the display 1 of the video display device 10 in Fig. 1 and connected components (GPU 2, etc.).

[0040] In order to determine the recommended scenery, as shown by arrow (1), data 21 (DATA-A) required for selecting the recommended scenery is sent from the video recording and playback device 10 to the server 11 via the Internet communication 12. As shown by arrow (2), the server 11 accesses the Internet 12 to collect necessary data 22 (DATA-B) other than the data 21 (DATA-A), and determines data 23 (DATA-C) that is a candidate for recommended scenery by processing the information data 21 and 22 as input using a predetermined recommended scenery selection algorithm or AI, and sends the data 23 (DATA-C) to the video recording and playback device 10 along arrow (3).

[0041] The data 21 (DATA-A) transmitted from the video recording / playback device 10 to the server 11 includes the following information DA1 to DA5.

[0042] DA1: User information The term information DA1 "user information" used in this specification refers to information about the user currently viewing the display 1 of the video recording / playback device 10. The video recording / playback device 10 may be provided with a user registration function, and users may be managed by the video recording / playback device 10 and the server 11. It is possible to know which user is using which video recording / playback device.

[0043] A video recording / playback device 10 used by a single user sends information about the currently logged-in user to the server 11. A video recording / playback device 10 used by multiple users may send information about the currently logged-in user, or may analyze images from the camera 8 of the video recording / playback device 10 and send information about the user in front of the video recording / playback device 10 to the server 11.

[0044] DA2: View history, operation history The term "scenery viewing history, operation history" as information DA2 used in this specification refers to historical information such as which scenery the user has viewed in the past and what operations he or she has performed. The history information is sent to the server 11. The history information is used by the server 11 as data for inferring the user's tastes and preferences and outputting recommended scenery candidates. The history information may be sent to the server 11 each time a scenery is viewed or an operation is performed, or may be sent all at once when a recommended scenery is selected.

[0045] For example, for a user who often views seascapes, the history information may be used to increase the proportion of seascapes or summer scenery displayed when inferring recommended scenery candidates. For example, in the case of animal scenery, for a user who often performs an operation to quickly change scenery, the history information may be used to decrease the proportion of scenery that includes animals when inferring recommended scenery candidates.

[0046] DA3: User state captured by camera, microphone, or sensor The term information DA3 "user's state acquired by a camera, microphone, or sensor" used in this specification is information inferring the user's state from data from the camera 8, microphone 9, and sensor (not shown) of the video recording and reproducing device 10. This may be used to output a scene candidate that matches the user's state when inferring a recommended scene candidate. For example, if it can be inferred that the user is concentrating while looking down at his / her desk, a "calm scene that allows concentration" may be output as a scene candidate. For example, if it can be inferred that the user is having dinner or a party with multiple people, a "lively and bright scene" may be output as a scene candidate. For example, if it can be inferred that the user is at a jazz bar or quietly drinking, a "night view scene" may be output as a scene candidate. For example, if it can be inferred that there is no one around, a "quiet scene" may be output as a scene candidate, and if necessary, the video recording and reproducing device 10 may be instructed to sleep.

[0047] DA4: Status of the installation environment obtained using cameras, microphones, sensors, or wireless signal conditions The expression "information DA4" used in this specification, "state of the installation environment acquired using a camera, microphone, sensor, or wireless radio wave conditions" refers to information about the environment in which the video recording and reproducing device 10 is placed, which is acquired and inferred using the camera 8, microphone 9, other sensors (not shown), wireless radio wave conditions, etc. of the video recording and reproducing device 10. This information is used to output scenery candidates that match the user's state when the server 11 infers recommended scenery candidates.

[0048] The information on the environment acquired using the wireless radio wave conditions and inferred based on the information is, for example, information obtained by using a method of measuring the amount of activity by utilizing phenomena such as deterioration of the radio wave reception sensitivity in the environment and loss of connection to the router when a large number of people own wireless devices such as smartphones and PCs. Information on the "wireless radio wave conditions" based on the SSID (connection name) and number of routers can be used to estimate the environment in which the routers are installed in the case of an environment such as an office district, based on the characteristic that there are a large number of routers (SSIDs). The wireless reception sensitivity, the status of the SSID of the router, etc. may be periodically saved. The saved information may be used to estimate the amount of activity or the environment around the router. The information may be used to estimate the scenery. For example, if it is estimated that there is a lot of human activity, a bustling scene such as a street or an amusement park may be displayed. For example, if it is estimated that there is a lot of business district, a quiet forest may be displayed.

[0049] For a quiet and calm environment, a "quiet and calm scene" may be output. For a busy environment, a "scenery that catches your eye at first sight" may be output. If an office environment is inferred from the camera 8, wireless signal conditions, etc., a "scenery that makes it easy to concentrate on work" may be output. If a home is inferred from the camera 8, wireless signal conditions, etc., a scene candidate for a "relaxing scene" may be output. If an environment with the lights off is inferred, a command to put the video recording and playback device 10 to sleep may be output.

[0050] The sensor used to estimate the environment of the video recording / playback device 10 may be a thermometer / hygrometer, barometer, altimeter, geomagnetic sensor, illuminometer, LIDAR, odor sensor, atmospheric sensor, etc., provided inside and / or outside the video recording / playback device 10. The wireless radio waves may be used to grasp the positional relationship with a router by Wi-Fi positioning, or to transmit environmental information to the video recording / playback device 10 by a beacon (such as Bluetooth), RFID, etc.

[0051] DA5: Location information of installation location The term "location information of the installation location" as used in this specification refers to location information of the installation location of the video recording and reproducing device 10, which is obtained from user settings, a GPS sensor, base station information from the SIM of the network I / F, and the like. The information may be information on the location of the video recording and reproducing device 10, which is obtained by estimation. The information DA5 is used to output a scene candidate that matches the user's state when the server 11 infers a recommended scene candidate. For example, when the server 11 determines the position of the sun on the spot based on the latitude and longitude of the installation location of the video recording and reproducing device 10, the server 11 may output a scene candidate that matches the position of the sun (sunrise scene at sunrise, daytime scene during the day, dusk scene at sunset, scene after sunset or nighttime, etc.). When the season of the installation location is determined from the latitude and longitude and the date and time, the server 11 may output a scene candidate that matches the determined season (summer scene in summer, etc.). When the country or region of the installation location is confirmed from the latitude and longitude, scenery candidates that correspond to the confirmation may be output (using calendar information, if the installation location of the device 10 is Japan, scenery related to days when there are events unique to Japan (such as Hinamatsuri) may be output).

[0052] The data 22 (DATA-B) that the server 11 collects from the Internet 12 includes the following information DB1 to DB9.

[0053] DB1:Date and time The term "date and time" in the information DB1 used in this specification refers to information on the date and time when a recommended scenery is requested by operating the video recording and playback device 10. This information is used by the server 11 to output scenery candidates that match the date, time, season, etc. of the location where the video recording and playback device 10 is installed by combining it with the location information of the installation location of the DA5 described above and other information.

[0054] DB2: Seasons The term "season" in information DB2 used in this specification refers to information related to the season of each country or region. As described above in DA5 "location information of installation location" and DB1 "date and time", this information is used when the server 11 outputs scenery candidates that match the season of the installation location of the video recording and playback device 10 by combining it with the season of the installation location of the video recording and playback device 10. The "season" of the installation location of the video recording and playback device 10 is estimated from DA5 "location information of installation location" and DB1 "date and time", and is used when outputting scenery candidates related to the "season" of the installation location of the video recording and playback device 10 based on the estimated information, i.e., the seasonal information recorded on the Internet including the server. For example, if the video recording and playback device 10 is installed in Japan and it is January 1st, it is possible to display scenery related to New Year on the video recording and playback device 10.

[0055] DB3: Time information (sunrise and sunset, moon age, etc.) The term "time information (sunrise and sunset, age of the moon, etc.)" in information DB3 used in this specification refers to information related to the time of day for each country or region, such as the times of sunrise and sunset, age of the moon, etc. As described above in DA5 "location information of the installation location," this is used when server 11 outputs scenery that matches the time, for example, a sunrise scenery when the requested time of a recommended scenery candidate is sunrise, or a night scenery after sunset.

[0056] DB4:Weather The term "weather" in the information DB4 used in this specification refers to weather information at the location where the video recording / playback device 10 is installed. For example, when a command to display a recommended landscape of the video recording / playback device 10 is operated when it is raining, the information is used when the server 11 outputs a candidate landscape in which a rainy landscape is output and rain or raindrops are added to the landscape image by CG to give the impression of looking outside from an actual window.

[0057] DB5:Map information The term "map information" in the information DB5 used in this specification refers to map information that can be obtained from Google Maps or the like. As described above in DA5 "location information of installation location," this information can specify the installation location of the device 10 on a map by combining the location information of the video recording and playback device 10 with the information DB5, and collects information on the country or region and map or geographical features. Based on the collected information, the server 11 outputs scenery candidates that match the country or region where the device 10 is installed. For example, if a certain color or object is not culturally preferred in the country, scenery candidates that do not include the unpreferred scenery may be output.

[0058] DB6:Calendar Information The term "calendar information" in DB6 used in this specification refers to calendar information for each country or region that can be obtained from Google Calendar, etc. The server 11 keeps track of events in the country or region (Christmas, Hinamatsuri in Japan, etc.), and outputs scenery candidates that take the events into consideration when a recommended scenery output is requested.

[0059] DB7:SNS The term "SNS" in the information DB7 used in this specification refers to information on SNS of users, etc. The server 11 analyzes the information DB7 to infer the user's hobbies and preferences or global trends, and outputs scenery candidates that take into account the inferred information.

[0060] DB8:News The term "news" in the information DB8 used in this specification refers to information related to news in the location where the video recording and reproducing device 10 is installed or global news. Based on the information DB8, the server 11 outputs scene candidates that are in line with the times when the device 10 is in operation. For example, during the Olympics, more scene candidates related to sports or the host city are output.

[0061] DB9:Scenery Information The term "scenery information" in the information DB 9 used in this specification refers to information related to the shooting location or subject of each scene image. The server 11 uses this information as reference information when grasping the details or background of the scene image and estimating recommended sceneries.

[0062] Unchanging information among the above-mentioned information DA1 to DA5 and DB1 to DB9 may be stored in the server 11. This, for example, makes it unnecessary to transmit or collect information (data) again, and can reduce the processing time or load when selecting (inferring) recommended scenery.

[0063] 2, data 23 (DATA-C) indicated by arrow (3) showing the data flow from the server 11 to the video recording and playback device 10 is data on "recommended scenery candidates" and is output by the server 11 based on a predetermined algorithm for selecting recommended scenery. According to an embodiment of the predetermined algorithm for selecting recommended scenery, in addition to the embodiments described in relation to data 21 (DATA-A) and data 22, the following two methods can also be used.

[0064] In the first method, the server 11 regards "sceneries that have been viewed frequently" and "sceneries that have been set as favorites many times using the favorite setting function" as popular scenery for all users, and outputs the popular scenery preferentially as recommended scenery candidates.

[0065] In the second method, the server 11 sets the recommendation function to semi-automatic, has the user notify the user of the keywords of the scenery that the user wants to be recommended via the video recording and reproducing device 10, and preferentially outputs the recommended scenery candidates based on the notified keywords. For example, the keywords may be "spring" or "summer" if the user wants to see scenery of a specific season, "sea" or "mountain" if the user has a preference for the subject of the shooting, or "cat" or "tower" if the user has a preference for the object that is included in the scenery. A plurality of keywords may be specified. The relationship between them may be specified by "and" or "or." The keyword may be specified by "selecting from candidates with the video recording and reproducing device," "inputting a character string to the video recording and reproducing device," "inputting voice with a microphone," "having the camera read a QR code (registered trademark) with a keyword set," or the like. Alternatively, "having the camera read a similar scenery," or "setting the URL of the similar scenery in the video recording and reproducing device (manual input, voice input with a microphone, QR code (registered trademark) with a camera, etc.)," ​​and a scenery similar to the set similar scenery may be output as a recommended scenery candidate.

[0066] Multiple algorithms described in this disclosure, including the first and second methods, may be combined or prioritized.

[0067] In addition to being programmed as a fixed algorithm, AI technology may be used to realize the recommended scenery selection algorithm in the server 11 that outputs the recommended scenery candidates of data 23 (DATA-C). In particular, deep learning, in which data 21 and data 22 are input data and data 23 is output, is an example. By understanding the user's behavior when the recommended scenery is displayed on the device 10 (played for a long time, set as a favorite, quickly changed to another scenery, etc.), it is possible to estimate whether the displayed scenery was liked by the user. Therefore, by continuously learning using the estimation results as correct answer data, it is possible to improve the model and increase the quality of the recommended scenery function.

[0068] The recommended scenery feature can be implemented in the user interface (UI) at the following times:

[0069] Timing 1: User spontaneously switches scenery The timing when the user issues a command to change the scenery using an interface such as an operation button (wired or wireless, not shown) provided on each video recording and playback device 10, a remote controller (one of the external devices 13) of the video recording and playback device 10, gesture input using the camera 8, or voice input using the microphone 9.

[0070] Timing 2: Automatic landscape switching over time Timing of scenery switching at regular intervals and / or at specified times. Specification may also be made in conjunction with turning the power OFF and ON. For example, scenery switching is performed by turning the power ON at 8am, and the power OFF at 8pm, automatically switching scenery without user operation. The user can set the interval or time of switching, and functions to be performed in conjunction with switching, such as power OFF / ON, in the video recording / playback device 10 or server 11. The setting means is the same as for Timing 1 above, "User-initiated scenery switching."

[0071] Next, with reference to FIG. 3, a screen display that is processed and displayed on the display 1 of the video recording / playback device 10 will be described. The "scenery selection" and "scenery playback" of recommended scenes are independent and different processes. The process using the data files 21, 22 is "scenery selection", and the scene is not displayed on the screen during the selection process. When the selection process is completed and the next scene is determined by the server 11, the process moves to "scenery playback" for the next scene, and the scene is played back in order on the screen as shown in display steps a1 to a4 in FIG. 3(a). As described above, the scene video to be played back is stored in the ROM 4 in the video recording and playback device 10 and / or in storage in the server 11 on the Internet 12. When playing back a video on the Internet 12, streaming playback is performed while storing the cache in the ROM 4. When playing back continuously, cache playback is performed. Therefore, the "scenery playback" process does not require Internet communication.

[0072] The scenery determined in the "Scenery Selection" process is reproduced following the "Scenery Reproduction" process in the form of a unique code. The unique code may be, for example and without limitation, a numeric code, a character code, a URI, etc. By using the code, the scenery data stored in the server 11 or the scenery data in the cache can be identified and reproduced.

[0073] Specifically, when it becomes necessary to switch to the next scenery in display step a1 in Fig. 3(a), the video recording and reproducing device 10 performs a "scenery selection" process without changing the screen. When the next scenery to be reproduced is determined, the video recording and reproducing device 10 performs a "scenery reproduction" process using a code representing the determined scenery. Upon completion of this process, the process moves to display step a2. This process flow is repeated through display steps a2, a3, and a4, whereby display steps a1 to a4 are displayed.

[0074] In display step b1 of FIG. 3(b), a list of candidate scenery is displayed after the "scenery selection" process. The scenery list is created by the "scenery selection" candidates and their thumbnails (reduced images). In some aspects, the user can select a particular scenery from the list and display it. In some aspects, the user can display and play the list as a scenery playback list in order from display step b2 to b5. If there is a scenery that is not to the user's liking, a command for the scenery that is not to the user's liking may be sent to the server 11 via the video recording and playback device 10 to exclude the scenery that is not to the user's liking, and the server 11 may output the recommended scenery candidates again.

[0075] In the case of the second method mentioned above, “making recommendations semi-automatic and notifying the user of keywords for scenery they would like recommended,” recommended scenery is presented based on the notified keywords, as shown in Figure 3(c). In Fig. 3(c), supplementary information is added to the "list of candidate landscapes" in Fig. 3(b). For example, in display step c1, the title "Summer" is added. This allows the user to easily understand, for example, what criteria were used to select the candidate landscapes.

[0076] The supplemental information becomes the information of data 21 or data 22 in Fig. 2, and when the server 11 returns data 23 of the scenery candidates, what information of data 21 or data 22 was used to select the candidate is included as metadata (additional data to the data) in the data 23. Using this metadata, the video recording and reproducing device 10 can display an image based on the supplemental information, as in display step c1 in Fig. 3(c).

[0077] Thus, Fig. 3(c) shows an example in which the keyword "Summer" is specified. A list of recommended summer scenery is displayed in display step c1. The user may select a scenery to be played back from the list of scenery, or may play back the list of scenery in the display steps in order as a playlist, as in display steps c2 to c5.

[0078] Fig. 4 shows a simplified block diagram of a video playback device 10 according to another embodiment of the present disclosure. This device has a similar configuration to the video recording and playback device 10 of the scenic video recording and playback system 100 shown in Fig. 1, and the same components are designated by the same reference numerals.

[0079] The video playback device 10 includes a display 1, a GPU for displaying video on the display 1, a CPU, a ROM, a RAM, a sensor interface (Sensor I / F: consisting of USB, SPI, I2C, UART, CAN, etc.) 6 for attaching various sensors, and a network interface (Network I / F: consisting of Wi-Fi, Bluetooth, LTE, wired LAN, etc.) 7 for communicating with an external device (corresponding to the external device 13 in FIG. 1) or the Internet 12. The display 1 can be attached to a wall and used as a virtual window. The scenery video to be played back is present in a memory (ROM) 4 in the video playback device 10 and / or a storage on the Internet 12 (corresponding to the server 11 in FIG. 1). To play back video on the Internet 12, streaming playback is performed while storing a cache in the memory 4. As a result, cache playback is performed during continuous playback, and Internet communication is not required.

[0080] A dedicated chip 14 for implementing this embodiment is built into the video playback device 10. The dedicated chip 14 incorporates the following chips required for the functions to be implemented: Video decoder chip: A chip that specializes in video decoding to reduce the load of real-time video processing. This chip reduces the load of decoding on the CPU or GPU. An H.265 decoder chip is a suitable example. Image processing chip: A chip that specializes in image processing of video frames in order to reduce the load of real-time video processing. This chip can reduce the image processing load on the CPU or GPU. Depending on the need, a dedicated image processing hardware accelerator, programmable accelerator, DSP, dedicated image processing CPU, etc. may be used. AI chip: A dedicated chip for performing AI processing in video processing to reduce the load of real-time video processing. AI chips are mainly used for inference processing. If learning is also performed within the device 10, a learning chip is built in.

[0081] FIG. 5 shows a general diagram of a normal processing flow in the video playback device 10. Video data 26 acquired from the memory (ROM) 4 or the Internet 12 is subjected to a decoding process 22 according to the format of the video data 26, and images or audio are extracted for each frame F1, F2, F3, F4.... The decoding process 27 is performed by the CPU 3 or a dedicated chip (such as an H.265 decoding chip) 14. The frame images are stored in the video memory 28 and displayed on the display 1. The process of displaying the images in the video memory 28 on the display 1 is basically performed by the graphics processing unit (GPU) 2. This frame display is repeated in accordance with the frame rate of the video (24 fps, 29.97 fps, 60 fps, etc.) (for example, at 24 fps, 24 frame images are displayed per second), and audio is played in accordance with the repeated display, thereby playing back the video and audio.

[0082] The video memory 23 is typically the RAM 5 or memory built into the GPU 2. Alternatively, if preferred, video memory built into the dedicated chip 14 may be used.

[0083] Fig. 6 shows a modified processing flow diagram in which two image processing steps 29, 30 are added to the normal processing flow in the video playback device 10 of Fig. 5. The frame image resulting from the decoding 27 is not sent directly to the video memory 28 as in the flow of Fig. 5, but is first saved and stored in the RAM 5 and subjected to a first image processing step (Pro1) 29. This image processing is performed by the CPU 3, or, when it is desired to reduce the CPU load, by a dedicated chip (image processing chip, AI chip) 14. The image that has undergone the image processing is sent to the video memory 28, and immediately before the processed image is displayed on the display 1, the GPU 2 performs a second image processing step (Pro2) 30.

[0084] In Fig. 6, as an example, the first image processing 29 binarizes the frames and removes fast-moving objects, for example, removing the birds in frames FR1, FR2, etc., and the second image processing 30 presents a rainy landscape seen through a window on the display 1 (displaying raindrops falling on the window). If desired, the sound of the video may also be processed or corrected in conjunction with the image processing.

[0085] In the first image processing 29, image processing is performed on the images of each frame FR1, FR2, ... using the CPU 3 or the dedicated chip 14. However, it is possible to not only perform general image processing such as filtering, but also to store past frame images by using the large-capacity RAM 5. Therefore, it is also possible to perform image processing using the transition of frame images (moving object detection, moving object removal, smoothing using multiple frames, etc.).

[0086] Information from other parts in the device 10 can also be used via the RAM 5. Therefore, for example, image processing using the brightness or color of the surroundings where the device is installed using information from the camera center, or image processing using information on the Internet is also possible. Other chips can also be used for image processing. So, for example, an AI chip can be used to process images, such as object recognition, face recognition, character recognition, and other image processing, and then image processing can be done according to the subject. For example, it can detect a person and blur the image so that the person cannot be identified. For example, it can recognize characters and automatically translate them.

[0087] The second image processing 25 described above performs image processing on the frame images f1, f2... that are the output of the first image processing 29, immediately before they are displayed on the display 1. However, essentially, it mainly performs filtering and other processing using information on the frame image alone immediately before it is displayed. Since the processing uses the pipeline of the GPU2, a variety of processing can be realized programmatically using shaders. In addition to image processing on the frame images, it is also possible to add other objects to the frame images as shown in Figure 6 (adding raindrops in Figure 6).

[0088] In the image processing of the first image processing 29 and the second image processing 30, any image processing can be performed. Specifically, image correction, image conversion, image processing of the frame image; compositing another image, two-dimensional computer graphics (2DCG), and rendered three-dimensional computer graphics (3DCG) with the frame image; and adding visual effects (VFX) to the frame image can be performed. Preprocessing such as image feature extraction and image recognition to realize these can also be performed. In some aspects, both the first image processing 29 and the second image processing 30 can perform all image processing. However, in some aspects, a higher frame rate can be maintained by performing image processing in parallel using two image processings in combination. In some aspects, processing efficiency can be improved by assigning suitable processing to each processor that performs the image processing of the first image processing 29 and the second image processing 30. For example, image correction, which performs the same processing on all pixels on a frame, can be processed faster by assigning it to the GPU 2 as a processor than by assigning it to the CPU 3. For example, in the case of AI processing, if an AI chip is mounted on the dedicated chip 14, high-speed processing can be achieved by assigning it to the AI ​​chip as a processor.

[0089] The image processing of this embodiment is not merely an image processing, but is characterized in that it targets scenery images. In order to "play back scenery as a beautiful view from a window or a view that the user likes", image processing is performed on scenery images in real time using the implementation means as in the above-mentioned embodiment. More specifically, the contents of the image processing are as follows.

[0090] <General color correction for video> Depending on the camera or shooting environment, the color tone of the entire landscape image may not be favorable. For example, the balance of the tone curve may be poor and lean to one side, or the white balance may not be appropriate. This image processing can automatically perform level correction, contrast correction, or color correction to correct the color tone of the entire image to a more favorable one.

[0091] Figure 7(a) shows an example of color correction of a typical landscape image on display 1. The original landscape image is dark overall, and the tone curve is leaning towards 0. By automatically correcting this so that the tone curve is balanced, it can be converted into a more pleasing color tone.

[0092] Since users have preferences regarding color tones, the video playback device 10 is configured to allow the user to change the presence, strength, or characteristics of this automatic correction according to user settings. Changes can be made in stages according to the user's knowledge of image quality, as follows: (1) Individually and precisely set brightness, contrast, etc. (2) Set the general tone level (e.g., "soft," "vivid," etc.) (3) Select a landscape with your preferred color tone In some aspects, the preferred color tone of the user may be inferred from scenery viewed in the past, and the preferred color tone inferred at the time of setting may be displayed or notified. This can contribute to the selection of the preferred color tone of the user, for example.

[0093] <Color correction according to the content of the landscape> (1) In color correction according to the content of a landscape, color correction suitable for the landscape is performed based on the subject of the landscape, the shooting time, etc. For example, the following color correction is performed. (2) If the landscape mainly contains forests, correct the image to make the green more prominent. (3) If it is a seascape, correct it so that the blue of the sea and / or the white of the clouds are more prominent. (4) If it is a sunset scene, do not correct the color level too much, but correct it so that the sunset looks red like the one seen with the naked eye. (5) If the landscape mainly shows autumn leaves, correct the image so that the red or yellow colors come out more prominently.

[0094] An example is shown in Figure 7(b). The original landscape image is recognized as an ocean scene, and the color tone is automatically corrected so that the blue of the ocean and the white of the clouds become clearer. In some embodiments, the entire frame image may be corrected at once. Alternatively, in some embodiments, a mask may be applied and only a portion of the image may be corrected (for example, only the clouds in Figure 7(b) are corrected).

[0095] To recognize the content of the scenery, the metadata of the scenery video (which stores information such as the shooting location, shooting time, and supplementary information about the scenery) and / or the content of the scenery grasped by image recognition (such as estimation from the content of similar scenery that appears in an image search, or inference using a scenery identification model (deep learning model) that has been trained to read a scenery image and reveal its shooting location, shooting time, etc.) may be used. As with the above-mentioned "general color correction for images," since there are user preferences, the presence or absence of this correction or its strength may be changed in the user settings.

[0096] <Image processing according to the weather or environment of the device installation location> When displaying a landscape image such as "a beautiful view from a window," the image may be corrected according to the weather at the location where the image reproducing device 10 is installed. This makes the device feel more like a window, for example, and creates the illusion that the landscape is actually nearby.

[0097] For example, if it is raining and a sunny scene is being played back on device 10, it is difficult to sense device 10 as a window. However, by adding an effect that makes it appear as if raindrops are on the window, as in Figure 8(a), it is possible to sense device 10 as a window.

[0098] Similarly, the sense of a window can be enhanced by adding a correction or effect appropriate to the weather, such as sunny or cloudy. Image processing or correction can also be performed based on environmental information other than the weather, such as "lowering the contrast if the surroundings of the device are dark" or "lowering the level of red if strong red is not preferred in the country where the device is located." Information collected from the Internet 12, information obtained from a sensor connected to the device 10 by wire or wirelessly, etc., can be used to grasp the weather. As with the above-mentioned "general color correction for an image," since there are user preferences, the user may change the presence or absence or strength of image processing according to the weather or environment in the user settings.

[0099] <Image processing that makes the scenery look like it is seen through a special window> The same landscape may look different when viewed through a frosted or stained glass window. The glass of the window acts as a filter, and the landscape may be perceived as looking different. Images obtained by subjecting a landscape to filter processing, which is image processing such as grayscaling, binarization, sepia conversion, cartooning, painting, or ink painting, may be perceived as looking different through a window. This can greatly change the impression of the landscape, allowing a different impression to be enjoyed. Depending on the user, a specific device, or other settings, image processing may be performed on the landscape image as if the landscape were viewed through a special window.

[0100] Fig. 8(b) shows an example in which a predetermined filter process has been applied to a landscape image. An image P2 of "landscape seen through a frosted window," an image P3 of "landscape seen through a stained glass window," and an image P4 of "landscape seen through a sepia-colored window" are displayed in addition to an original landscape image P1. By operating the image playback device 10, the user can specify a special window or its features (such as the degree of frosting in the frosted glass, the size of the mesh in the stained glass, etc.) according to preference.

[0101] According to this embodiment, the video playback device may have a virtual window type display that displays an image as if the scenery were reflected in a window. A step is provided for combining a predetermined preferred glass window image with the scenery image to display an image such as "scenery seen through a frosted glass window", "scenery seen through a stained glass window", "scenery seen through a sepia window", etc.

[0102] As another embodiment of the present disclosure, FIG. 9 shows an example of adding spatial information to a video. FIG. 9(a) shows an original video displayed on the display 1 of the video playback device, FIG. 9(b) shows a video to which spatial information has been added, and FIG. 9(c) shows a video to which new information has been added. The video playback device of this embodiment has a display 1 that displays a desired landscape, a unit that sets spatial information of the landscape, a unit that receives a notification from the outside, a unit that maps the notification from the outside to an object that can be inserted into the image space, and a unit that dynamically inserts an object into the video. The video playback device 10 used in FIGS. 1 to 8 may be used. In FIG. 9(b), spatial information, which is a three-dimensional model related to perspective, is set in advance for the landscape image.

[0103] Figure 10 shows an example in which when some event occurs outside the video playback device, such as an incoming email or a notification on a social networking site, the event is converted into an object that can be inserted into the video model and dynamically inserted into the video.

[0104] The above conversion from an event to an object is performed using, for example, the following rules, but may be individually set by the user. (a) Manual binding (converting a taxi notification to a taxi object). (b) Automatically converting service notifications with blue as the service color into blue animals, etc. (c) For notifications that should remain (such as a taxi arrival), place an object that remains, and for notifications that can disappear (such as SNS notifications), place an object that disappears (such as an animal).

[0105] FIG. 9(c) shows an example of a reserved taxi waiting according to the rule (a). After informing the user of the above rules in advance, an object that is not usually present in the scenery is generated in the scenery, which allows the user to recognize it as a notification. In FIG. 10(a), scenery 31 is displayed on the display 1. As in FIG. 9(b), spatial information is added to the video. When an email arrives from outside the playback device, the incoming notification is mapped to an object that can be inserted into the video space of the spatial information, as shown in FIG. 10(b). A video of a bird object 32 flying with a letter in its mouth is superimposed on the scenery 31. This allows the user to be notified of the incoming email naturally without disturbing the scenery 31. The notification may include not only the superimposition of the object but also a predetermined ringtone. The incoming email may be an incoming call to the video playback device or a mobile device such as a smartphone. The object may be configured to indicate that the email is received by the mobile device. The object 32 may be a video of a bird flapping its wings or a still image. The dynamic insertion of the object 32 may be selectively set to at least one of fade in, slide in, and slow in. In the example of Fig. 10, the addition of spatial information to the scenery information 31 is not limited to three-dimensional information, but also includes two-dimensional information.

[0106] According to this embodiment, by synthesizing and displaying a dynamic or static object indicating a notification with a landscape image, it is possible to notify a user viewing the landscape image without impairing the comfort of the user. This embodiment is not limited to a method of displaying a landscape image, but is also applicable to a method of displaying an image other than the landscape.

[0107] As another embodiment of the present disclosure, Fig. 11(a) shows an example in which a landscape is displayed on the display 1 when a person M is standing in front of the image display device 40, and Fig. 11(b) shows an example in which a landscape is displayed on the display 1 when a person M is standing diagonally in front of the image display device 40. Fig. 11(c) shows a camera view image of the person M captured by the camera sensor device 41 provided in the image display device 40 in the state of Fig. 11(a), and Fig. 11(d) shows an image captured by the camera sensor device 41 in the state of Fig. 11(b). Figs. 12(a) and 12(b) show CG (computer graphic) images created by an image processing unit built into the image display device 40, corresponding to the landscape images of Figs. 11(a) and 11(b), respectively.

[0108] The image display device 40 includes a display 1, a camera 41 that recognizes the position of the viewer M in front of the display 1, and a computer graphic generation unit that generates an image of the display 1, and is configured to display images of scenery, etc., as real-time CG rendering (image output calculated by a program in real time) or real-time images from a live camera on the display 1. The devices used in the above embodiments can be used. An image display is possible in which the display image moves in conjunction with the movement of a specific part of the viewer M.

[0109] In Fig. 11(a), the camera sensor device 41 can be used for face tracking to extract the eyes or outline of a person M in front. The camera sensor device 41 may include a depth sensor. In this embodiment, when the person M moves to the right as shown in Fig. 11(b), the image displayed on the display 1 changes in a manner similar to the change in the scenery seen through a window in the real world. For example, something that was not visible from the front (image P2) becomes visible to the person M. As the viewing angle of the person M changes, the visibility (perspective) in the image also changes.

[0110] By the movement of person M described above, the image on display 1 can be displayed as if seen live through a window-shaped display 1. To achieve this, it is necessary to calculate the change distance d and change angle α of the camera (or live camera) 42, which performs real-time rendering inside the CG, from the front position in FIG. 12(a) to the tilted position in FIG. 12(b). To achieve this, it is first necessary to estimate the person's movement distance x and the distance z from the camera sensor device 41 to the person's face (or eyes) when person M moves from the position in FIG. 11(a) to the position in FIG. 11(b). Images p1, p2, and camera 42 in FIG. 12 correspond to images P1, P2, and camera sensor device 41 in FIG. 11, respectively.

[0111] First, if the camera sensor device 41 is equipped with a depth sensor and can measure the distance z, the depth sensor is used. If the camera sensor device 41 is not equipped with a depth sensor, the distance z is calculated by comparing the eye size m extracted by face tracking with half the length L of the image captured by the camera sensor device 41 in Figs. 11(c) and (d). The eye size may be defined as the length from the outer corner to the inner corner of the eye (eye opening). In this case, since the calculated distance differs depending on the lens angle of view of the camera sensor device 41, the coefficient k of the camera sensor device 41 is calculated in advance so that the distance z calculated from m and L is the same as the actually measured distance. The moving distance x can be calculated by calculating the moving distance l from the center line of the image captured by the camera 41 shown in Fig. 11(D) to the center of both eyes and adding the coefficient k. Next, the angle θ from the center of the image display device 40 to the center of both eyes after the person M has moved can be calculated as θ=arctanx / z from tanθ=x / z.

[0112] 12(b), when the movement distance of the camera inside the CG is limited to D, the movement distance d of the camera 42 inside the CG is calculated as d=D / Lxl. The angle α can be calculated as α=jxθ by considering the coefficient j that matches the camera characteristics of the CG (or live camera) for the angle θ.

[0113] Based on the calculated angle α and moving distance d, the visible range of the image displayed on the device 40, that is, the perspective (field of view), can be changed in real time in response to the movement of the person M. This makes it possible to obtain a natural experience, for example.

[0114] The live camera captures the outdoor scenery from indoors through a window. As explained in FIG. 1, the images captured by the camera are constantly uploaded to a server and displayed on a display. The live cameras are installed in various parts of the world, and the coefficient j changes depending on the characteristics of the camera, such as the angle of view. The live camera moves and rotates in conjunction with the display camera 41, changing the perspective.

[0115] Figures 12(a) and (b) are CG images. In the CG images that obtain an image as if the camera 42 had moved, the dotted lines radiating from the virtual camera 42 correspond to the image captured by the camera, which is displayed on the screen in Figures 11(a) and (b). When generating the CG, the virtual camera 42 is moved and the above calculations are performed.

[0116] According to this embodiment, an image stored inside the image display device 40 or an image obtained from a live camera installed outside is output in CG (for example, the image in FIG. 12(a)). When person M is in a specified position, an image limited to a specified frame (for example, image P1 in FIG. 11(a)) is displayed on the display 1. The image moves in conjunction with the movement of person M, and the hidden image P2 appears. This allows the user to have a natural experience in which the range visible through a window changes as if, for example, they were peering through it.

[0117] According to this embodiment, the image display function in which the visible range of the scenery and perspective change in conjunction with face tracking can be realized by linking the position and angle of a camera such as a real-time rendering camera in CG (computer graphics) or a live camera to the person's line of sight in real time.

[0118] In this embodiment, the person M moves sideways parallel to the display 1. The camera 42 may be moved vertically in accordance with the movement of the person M in the vertical direction of the display 1. When the person M approaches the display, the camera 42 also approaches the frame. It is possible to display a wide-angle field of view including not only the image p1 but also the image p2, and when the person M moves away, it is possible to display a narrow-angle field of view centered on the image p1. By moving the camera 42 in conjunction with the movement of the person M in other directions, it is possible to vary the field of view of the scenery seen through the frame F1. The viewer M can experience seeing the outside scenery as if the display 1 were an actual window. The display 1 in FIG. 11(a) may be set to a default state, and when no person is present, a standby state may be displayed, or when the person leaves, a termination state may be displayed.

[0119] In this embodiment, the configuration of the image display device is shown in which the viewer's face movement is tracked to link the image and change the viewable range and perspective, but the present invention is not limited to this. The movement of other parts of the body (for example, the head or the entire body) may also be tracked.

[0120] As another embodiment of the present disclosure, Fig. 13 shows an example of cutout display using position detection of multiple displays 51, 52, 53 and synchronous playback of multiple displays. Fig. 13(a) shows an example of cutout display using position detection of multiple displays, and Fig. 13(b) shows an example of synchronous playback of multiple displays 51, 52, 53. Displays 51, 52, 53 may include display 1 in Fig. 1 and Fig. 3, video recording and playback device 10 having image processing, storage and playback functions such as computer graphics, etc. Although an example of three displays will be described as multiple displays, the number of displays is not limited to three, and may be two or four or more.

[0121] Fig. 13(a) shows an example in which the image size or position to be cut out is determined using the relative distance between the display devices. In the cutout display method using position detection of multiple displays, at least one of the following methods can be used to obtain the cutout position. Method A: The positions of displays 51, 52, and 53 are detected from an image captured by an external electronic device equipped with a camera, and the cutout positions of the displays, enclosed by dashed lines in FIG. 13(a), are calculated. Method B: Using the markings as an aid to method A above, calculate the cutout position of the display, similarly surrounded by a dashed line. Method C: The positions of the displays 51, 52, and 53 are detected based on the distances moved from their initial positions, and the cutout positions are calculated.

[0122] As shown in FIG. 13(b), when a cut-out image is displayed on multiple displays 51, 52, and 53, they need to be synchronized. In the display example of FIG. 16, as described later, playback position information of a leader display among the multiple displays may be transferred to other follower displays via communication such as Bluetooth to achieve synchronization. In some aspects, the multiple displays may be synchronized with each other. In data transfer by communication, the display can be synchronized with only the difference in communication delay time (latency) that occurs from when a transfer request is issued until the data is actually sent.

[0123] Regarding the cut-out display using position detection of the multiple displays 51, 52, and 53 in FIG. 13(a), a method of calculating the cut-out position from the positions of the multiple displays 51, 52, and 53 will be described.

[0124] The physical positions of multiple displays 51, 52, 53 installed on a wall or the like are measured by any one of the above methods A, B, or C, or by another method of position detection. The sizes of the multiple displays may be the same or different.

[0125] The display positions may be detected from the image acquired by the electronic device of method A, and the cutout position may be calculated. First, as shown in FIG. 14(a), an image is acquired by a camera of a smartphone, tablet, PC, or the like. An image including three displays 51, 52, and 53 installed on a wall or the like is acquired using an electronic device capable of processing. Next, the physical positions of the three displays, such as the relative distance between the displays in the image, are detected from the acquired image using a method such as display pattern recognition. The electronic device calculates the image cutout position from the acquired position information. Position information for cutting out the image is transmitted to each display.

[0126] 14(b), a method B for calculating the relative distances between displays 51, 52, and 53 using markings will be described as a supplement to the above method A. As a supplement to the display position detection method of method A, a dot or a distinctive symbol K or the like can be attached to a wall or displayed on a display and used as a marking. The electronic device detects the physical positions of the multiple displays more accurately by using the markings in the image acquired by the camera.

[0127] Method C for detecting a physical position from a moving distance from an initial position will be described with reference to Fig. 15(a). First, as shown in Fig. 15(a), for example, three displays 51, 52, and 53 are placed in a predetermined initial position state, for example, by lining them up horizontally with their vertical centers aligned, and each display stores its own initial relative position.

[0128] Next, as shown in Fig. 15(b), each of the displays 51, 52, and 53 is moved from the initial position state to a desired position. In this case, each of the displays 51, 52, and 53 may be configured to be equipped with an acceleration sensor and to detect the distance of movement from the initial state by, for example, detecting acceleration to calculate the distance of movement. The relative positions of each of the displays can be calculated from the detected distance of movement. The position of the image to be cut out can be calculated from the stored initial position and the position after movement, and the cut-out image can be displayed.

[0129] In this embodiment, the size of the image can be set arbitrarily when calculating the image cutout position using position detection methods A, B, and C. The relative positional relationship of the cutout position does not change by changing the magnification of the image to be cut out while maintaining the relative positions and size ratios of the frames F1, F2, and F3 enclosed by dashed lines in Fig. 16(a). This is used to calculate the cutout position. By sharing the calculated cutout position on multiple displays, the cutout image can be displayed as shown in Fig. 16(b).

[0130] Next, a method of synchronizing and playing back a plurality of displays 51, 52, and 53 will be described with reference to FIG. 16(b). When the display medium is video, each display needs to synchronize the playback positions of the video with each other while displaying a cut-out image. For this purpose, for example, one display 52 out of the plurality (three displays) 51, 52, and 53 is set as a leader. The leader display 52 transfers its own playback position information to the other plurality of follower displays 51 and 53 via communication such as Bluetooth. When the follower displays 51 and 53 receive the playback position information from the leader display 52, they adjust the playback position of the video to the leader's playback position while displaying the same video cut out on each display. The leader display 52 periodically sends playback position information, so that the leader display 52 can play back videos synchronized with the follower displays 51 and 53 with only a communication delay as a difference. In other words, the video playback position of the leader display is shared with the follower displays, and synchronized videos can be played back.

[0131] The image display system of this embodiment includes a plurality of arbitrarily arranged displays, an electronic device equipped with a camera for capturing an image including the plurality of displays, a position detection unit for detecting the positions of the plurality of displays in the captured image, and a display unit for cutting out an image from the detected positions of the plurality of displays and displaying the image on the display. The position detection unit and the display unit can be realized by the devices and computer graphic processing of Figures 1, 3, 5, and 6, as in the above-mentioned embodiments.

[0132] In this embodiment, multiple displays are arranged at positions preferred by the user, and an image of scenery is cropped by the frame of each display and displayed on each display. This allows the user to have a natural experience of viewing scenery as if a virtual window were installed on a wall, for example. In combination with the face tracking embodiment of Figs. 11 and 12, an image can be provided in which an image hidden in the blind spot of the display (virtual window) appears in conjunction with the viewer's movement in front of the display. By combining with other embodiments, for example, the image display system of this embodiment can provide the user with a more comfortable viewing experience. This embodiment is not limited to scenery images, and can also be applied to general images.

[0133] The embodiments of the present disclosure are illustrative in all respects, and the scope of the present disclosure includes all modifications within the meaning and scope of the claims. For example, the landscape video recording and reproducing system is not limited to a video recording and reproducing system, but includes an image reproducing system, device, and method without a recording function, and the image display device includes an image recording device, system, and method.

[0134] Although some embodiments and examples of the present disclosure have been described above, these embodiments and examples are illustrative of the present disclosure. For example, the above-mentioned embodiments have been described in detail to clearly explain the present disclosure, and dimensions, configurations, materials, and circuits may be added and changed as necessary. In addition, embodiments in which one or more features of the present disclosure listed above are arbitrarily combined are also included in the scope of the present disclosure. The claims encompass many modifications to the embodiments without departing from the technical idea of ​​the present disclosure. Therefore, the embodiments and examples disclosed in this specification are presented for illustrative purposes and should not be considered as limiting the scope of the present disclosure. Explanation of symbols

[0135] 1. Display 2. GPU 3. CPU 4. ROM 5. RAM 8. Camera 9. Microphone, etc. 6…Sensor interface 7…Network interface 10...Video recording and playback device 11…Server 12. Internet 13…External equipment 14…Special chip 21…DATA-A 22…DATA―B 23…DATA―C 24,25…Image processing 26...Video data 27…Decoding process 28…Video memory 29…First image processing 30…Second image processing 31…Landscape 32…Object 40...Image display device 41...Camera (sensor device) 42...Camera, Live Camera, Virtual Camera 51, 52, 53…Display 100...Landscape video recording and playback system a2, a3, a4...Display steps b2, b3, b4, b5...Display step c1, c2, c3, c4, c5...Display steps M...people P1, P2, P3, P4...Video p1, p2...image

Claims

1. An external input interface; A memory for storing image data; A processing unit; A graphics processing unit; A plurality of displays for displaying images; a user-friendly setting mode, the user-friendly setting mode being an image quality improvement mode; and an internet interface for reading video data stored on the internet; A decode chip and An image display device having a video memory, The image display device includes: Reading the video data stored in the memory and / or the Internet; The read video data is decoded by the decode chip in accordance with the format of the video data; storing the frame images decoded into each video frame in the video memory; Detect the positions of multiple arbitrarily placed displays, Crop the frame image from the detected display position; The cut-out frame images are synchronously displayed on a display at the positions detected in sequence for each of the video frames; Setting one of the plurality of displays as a leader display and setting the other displays as follower displays; Detecting the positions of a plurality of arbitrarily arranged displays includes the following steps: the leader display acquires position information of the plurality of displays from the follower display; Image display device.

2. An external input interface; A memory for storing image data; A processing unit; A graphics processing unit; A plurality of displays for displaying images; a user-friendly setting mode, the user-friendly setting mode being an image quality improvement mode; and an internet interface for reading video data stored on the internet; A decode chip and An image display device having a video memory, The image display device includes: Reading the video data stored in the memory and / or the Internet; The read video data is decoded by the decode chip in accordance with the format of the video data; storing the frame images decoded into each video frame in the video memory; Detect the positions of multiple arbitrarily placed displays, Crop the frame image from the detected display position; The cut-out frame images are synchronously displayed on a display at the positions detected in sequence for each of the video frames; Setting one of the plurality of displays as a leader display and setting the other displays as follower displays; Transferring playback position information of the leader display to the follower display; When the follower display receives the playback position information from the leader display, the follower display cuts out and displays the same frame image on each of the plurality of displays. Image display device.

3. An external input interface; A memory for storing image data; A processing unit; A graphics processing unit; A plurality of displays for displaying images; a user-friendly setting mode, the user-friendly setting mode being an image quality improvement mode; and an internet interface for reading video data stored on the internet; A decode chip and An image display device having a video memory, The image display device includes: Reading the video data stored in the memory and / or the Internet; The read video data is decoded by the decode chip in accordance with the format of the video data; storing the frame images decoded into each video frame in the video memory; Detect the positions of multiple arbitrarily placed displays, Crop the frame image from the detected display position; The cut-out frame images are synchronously displayed on a display at the positions detected in sequence for each of the video frames; Setting one of the plurality of displays as a leader display and setting the other displays as follower displays; the leader display cuts out a frame image to be displayed by the leader display and a frame image to be displayed by the follower display based on the acquired position information of the other displays; The cropped frame image is sent to the follower display; The follower display synchronously displays the cropped frame image. Image display device.

4. The positions of multiple arbitrarily arranged displays are detected from an image including multiple displays. The image display device according to any one of claims 1 to 3.

5. The image including the plurality of displays is an image including all of the plurality of displays. The image display device according to claim 4.

6. Detecting the positions of multiple arbitrarily arranged displays is setting the plurality of displays at initial positions, and then moving and installing the plurality of displays at installation locations, and detecting the distance of the moves; The image display device according to any one of claims 1 to 5.

7. A random access memory for storing the decoded frame image is further provided. performing a first image processing on the frame image stored in the random access memory and storing the result in the video memory; performing a second image processing on the frame image stored in the video memory; displaying the frame images that have been subjected to the first and second image processing on the display in order for each of the video frames; The image display device according to any one of claims 1 to 6.

8. The display is a virtual window type display that displays an image as if the scenery were reflected in a window. The image display device according to any one of claims 1 to 7.

9. A recognition camera, A computer graphic generating function; the recognition camera recognizes the position of a viewer in front of the display; moving a display image on the display in conjunction with a movement of a predetermined part of the viewer; The image display device according to any one of claims 1 to 8.

10. Tracking the movement of a predetermined part of the viewer; setting a virtual camera of the recognition camera in accordance with a position of the viewer on computer graphics; A view of the scenery seen from the virtual camera is displayed on the display. The image display device according to claim 9.

11. a unit for setting spatial information of a scene displayed by at least one of the plurality of displays; A unit for receiving notifications from an external device; a unit for mapping the external notification to an object insertable in a video space; a unit for dynamically inserting objects into said scene image; Equipped The image display device according to any one of claims 1 to 10.

12. The dynamic insertion of the object is selectively set to at least one of fade-in, slide-in, and slow-in. The image display device according to claim 11.

Images