Projection device and operation method thereof

Abstract

A projection device according to an embodiment includes: a projection unit; a memory for storing one or more instructions; and at least one processor for executing the one or more instructions, wherein the at least one processor may execute the one or more instructions to: obtain a first image; obtain spatial information of a screen and location information of a user; generate a 3D object image and a background image on the basis of the spatial information of the screen and the location information of the user; and control the projection unit to project a second image obtained by combining the first image, the 3D object image, and the background image.

Description

Projection device and method of operation thereof

[0001] The present disclosure relates to a projection device for projecting an image and an operating method thereof.

[0002] A projection device can refer to a device that projects an image onto a screen or space. Projection devices can include projectors, devices that provide virtual reality (VR), augmented reality (AR), or mixed reality (MR). Projection devices are utilized in various fields, and can be used, for example, for lectures or presentations in classrooms or conference rooms, or for projecting movies onto screens in movie theaters. Devices that provide VR, AR, or MR can provide the experience of watching a movie in a movie theater by displaying images on a screen (display) positioned near the user's eyes when the device is worn.

[0003] Recently, the use of projectors to create home theaters has become increasingly common, allowing users to enjoy movies or games on a large screen. When viewing video content using a projector, not only the content's high resolution and projection performance, but also the viewing environment is becoming increasingly important.

[0004] A projection device according to one embodiment may include a projection unit including a projection circuit, a memory storing one or more instructions, and a processing circuit, and may include at least one processor that individually and / or collectively executes the one or more instructions.

[0005] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to obtain a first image.

[0006] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to obtain spatial information of the screen and positional information of the user.

[0007] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to generate a three-dimensional object image and a background image based on spatial information of the screen and positional information of the user.

[0008] The one or more instructions, when individually and / or collectively executed by the at least one processor, may control the projection unit to project a second image that is a composite of the first image, the three-dimensional object image, and the background image.

[0009] A method of operating a projection device according to one embodiment may include a step of acquiring a first image.

[0010] A method of operating a projection device according to one embodiment may include a step of obtaining spatial information of a screen.

[0011] A method of operating a projection device according to one embodiment may include a step of obtaining location information of a user.

[0012] A method of operating a projection device according to one embodiment may include a step of generating a three-dimensional object image and a background image based on spatial information of the screen and positional information of the user.

[0013] A method of operating a projection device according to one embodiment may include a step of projecting a second image synthesized from the first image, the three-dimensional object image, and the background image.

[0014] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following detailed description considered in conjunction with the accompanying drawings.

[0015] FIG. 1 is a drawing showing an example of a projection environment according to one embodiment.

[0016] FIG. 2 is a drawing showing an example of a configuration of a virtual space image according to one embodiment.

[0017] FIG. 3 is a flowchart illustrating an example of a method of operating a projection device according to one embodiment.

[0018] FIG. 4 is a diagram illustrating an operation of a projection device according to one embodiment to obtain spatial information of a screen and location information of a user.

[0019] FIGS. 5 and 6 are drawings for explaining an example of a method for generating a background image and a 3D object image by a projection device according to one embodiment.

[0020] FIG. 7 is a drawing for explaining an example of an operation of determining a background image based on an image to be projected by a projection device according to one embodiment.

[0021] Figure 8 is a flowchart showing an example of a method of operating a projection device according to one embodiment.

[0022] FIG. 9 is a drawing for explaining an example of an operation of a projection device according to one embodiment to change a virtual space image as a user's position changes.

[0023] Fig. 10 is a flowchart showing an example of a method of operating a projection device according to one embodiment.

[0024] FIG. 11 is a drawing for explaining an example of an operation of a projection device according to one embodiment to change a virtual space image as time information changes.

[0025] FIGS. 12 and 13 are drawings showing examples of a projection device according to one embodiment providing various virtual space images.

[0026] FIG. 14 is a drawing for explaining an example of the operation of a projection device according to one embodiment when multiple users are recognized.

[0027] FIG. 15 is a drawing for explaining an example of an operation of a projection device according to one embodiment to project a virtual space image.

[0028] Fig. 16 is a block diagram showing an example of the configuration of a projection device according to one embodiment.

[0029] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0030] The terms used in this disclosure will be briefly explained, and the present disclosure will be described in detail with reference to the drawings.

[0031] The terms used in this invention have been selected from widely used, current terms, taking into account their functions. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, some terms have been arbitrarily selected, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the invention.

[0032] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein mean a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0033] Below, various embodiments of the present disclosure are described in detail with reference to the attached drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, portions irrelevant to the description may be omitted for clarity of description, and similar parts are designated by similar reference numerals throughout the disclosure.

[0034] In embodiments of the present disclosure, the term “user” may mean a person who controls a system, function, or operation, and may include a developer, administrator, or installer.

[0035] In the present disclosure, 'image' or 'picture' may refer to a still image, a moving image composed of a plurality of consecutive still images (or frames), or a video.

[0036] FIG. 1 is a drawing showing an example of a projection environment according to one embodiment.

[0037] Referring to FIG. 1, a projection device (100) according to one embodiment can project image content (50) and an image (10, hereinafter, a virtual space image) representing a virtual space onto a screen. At this time, the screen onto which the image (10) is projected can be configured in various forms. When the projection device (100) is a projector, the screen can refer to a physical space onto which the image content (50) is projected. For example, the screen can include a wall or a screen made of fabric.

[0038] If the projection device (100) is a device that provides VR, AR, or MR, the screen may refer to a display included in the projection device (100). For example, the device that provides VR, AR, or MR may be implemented in the form of a glasses-type wearable device that includes a head-mounted display (HMD) that can be mounted on the head, and may include a display. The display may be a transparent display or an opaque display. The device that provides VR, AR, or MR may output (display) image content (50) to the display.

[0039] Hereinafter, for convenience of explanation, an example is illustrated and described in which the projection device (100) is a projector, but the same can be applied even when the projection device (100) is a device that provides VR, AR, or MR.

[0040] A projection device (100) according to one embodiment can provide a user (viewer) with an experience of viewing video content (50) in various environments by projecting not only video content (50) but also a virtual space image (10) onto a screen.

[0041] For example, a virtual space image (10) may be projected onto a wall of a living room, as illustrated in Fig. 1. The virtual space image (10) may include a window, curtain, display device, etc., and may include a background image (20) of the sea, city, mountain, etc., shown through the window. However, the present invention is not limited thereto.

[0042] The projection device (100) can generate three-dimensional (3D) images (3D object images, 30) of objects such as windows, curtains, and display devices included in the virtual space image (10) according to the user's position and viewing direction. In addition, the projection device (100) can generate a background image (20) included in the virtual space image (10) according to the user's position and viewing direction, as if the user were actually viewing the ocean through a window. Accordingly, a sense of reality can be provided to the user.

[0043] A projection device (100) according to one embodiment can generate a final image to be projected by overlapping and synthesizing a 3D object image (30) with a background image (20) and synthesizing image content (50) onto a display device included in the 3D object image (30).

[0044] The projection device (100) can provide the user with an environment similar to watching video content (50) in a space with an ocean view by projecting the final image onto a screen.

[0045] FIG. 2 is a drawing showing an example of a configuration of a virtual space image according to one embodiment.

[0046] Referring to FIG. 2, the virtual space image (10) may include a background image (20) and a three-dimensional object image (30). For example, the background image (20) according to one embodiment may include an image representing a background such as the sea, the sky, a beach, a city, a mountain, or space.

[0047] A 3D object image (30) according to one embodiment may include an image representing objects such as windows, curtains, shelves, and chairs in 3D. The projection device (100) may apply a 3D effect according to the viewing direction of the user (70) to objects included in the 3D object image (30).

[0048] Additionally, the 3D object image (30) may include an object (e.g., a display) that displays image content (50). However, the present invention is not limited thereto.

[0049] A projection device (100) according to one embodiment can generate a virtual space image (10) by synthesizing a three-dimensional object image (30) and a background image (20). The projection device (100) can overlap the three-dimensional object image (30) on the background image (20).

[0050] A projection device (100) according to one embodiment can generate a final image to be projected by synthesizing a virtual space image (10) and image content (50).

[0051] Video content (50) according to one embodiment may include content for viewing by a user (70). For example, and not limited thereto, the video content (50) may include various content such as movies, TV programs, videos, motion pictures, online streaming services, and advertising content. However, the present invention is not limited thereto.

[0052] A projection device (100) according to one embodiment can generate a final image by overlapping image content (50) with a display object included in a virtual space image (10).

[0053] Referring to the drawings below, operations of a projection device (100) according to one embodiment generating a virtual space image (10) in accordance with the line of sight (viewing direction) of a screen and a user (70), synthesizing image content (50) onto the virtual space image (10), and projecting the synthesized image content will be described in more detail.

[0054] FIG. 3 is a flowchart illustrating an example of a method of operating a projection device according to one embodiment.

[0055] Referring to FIG. 3, a projection device (100) according to one embodiment can obtain an image (image content) to be projected (S310).

[0056] According to one embodiment, the image may be an image previously stored in the projection device (100) or an image received from an external device. However, the present invention is not limited thereto. In addition, the image according to one embodiment may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion have been performed in the video processing unit.

[0057] A projection device (100) according to one embodiment can obtain spatial information of a screen (S320).

[0058] According to one embodiment, a projection device (100) can acquire spatial information of a screen using at least one camera (e.g., an image sensor, a depth sensor, etc.). For example, the projection device (100) can acquire an image of the screen from the image sensor and can acquire depth information of the screen from the depth sensor. However, the present invention is not limited thereto.

[0059] According to one embodiment, a projection device (100) can receive an image of a screen captured and depth information of the screen from an external device. The external device can include at least one image sensor and a depth sensor. The external device can obtain an image of the screen captured using the image sensor and obtain depth information of the screen using the depth sensor. The external device can transmit the image of the screen captured and the depth information of the screen to the projection device (100). However, the present invention is not limited thereto.

[0060] The projection device (100) can obtain spatial information of the screen based on the captured image and depth information.

[0061] The spatial information of the screen may include the shape of the screen, distance information from the projection device (100) to the screen, position information of the screen, size information, position information for the corners (boundaries) of the screen, etc.

[0062] For example, the projection device (100) can recognize the corner of the wall based on the captured image and depth information, and, based on the recognized corner, identify the size and shape of the screen, the distance from the projection device (100) to the screen, etc.

[0063] According to one embodiment, a projection device (100) can obtain user location information (S330).

[0064] According to one embodiment, the projection device (100) can obtain user location information using at least one camera (e.g., an image sensor, an infrared sensor, etc.). For example, the projection device (100) can identify whether a user is present, the distance between the user and the projection device (100), the direction in which the user is located relative to the projection device (100), etc. However, the present invention is not limited thereto.

[0065] A projection device (100) according to one embodiment can generate a 3D object image and a background image based on spatial information of the screen and location information of the user (S340).

[0066] The projection device (100) can set a 3D object image and a background image based on user input. For example, the projection device (100) can set a 3D object image based on a user input selecting one of pre-stored candidate template images. The projection device (100) can pre-store candidate template images that differ in shape, size, position, etc. of windows included in the image.

[0067] The projection device (100) can set a background image based on a user input selecting one of the pre-stored candidate background images. The projection device (100) can pre-store candidate background images such as the sea, beach, mountains, and cities. A method for setting a 3D object image and a background image based on the user input will be described in more detail below with reference to FIG. 5.

[0068] The projection device (100) can set a background image based on the category information of the image to be projected.

[0069] According to one embodiment, a projection device (100) can adjust a 3D object image and a background image based on spatial information of the screen and location information of the user when a 3D object image and a background image are set.

[0070] For example, the projection device (100) can determine the sizes of the 3D object image and background image based on the spatial information of the screen. The projection device (100) can adjust the sizes of the 3D object image and background image based on the recognized corner of the wall so that the 3D object image and background image are projected on the entire wall. However, the present invention is not limited thereto.

[0071] In addition, the projection device (100) can adjust the depth information of objects included in the 3D object image based on the spatial information of the screen and the position information of the user. For example, when the position of the user changes, the viewpoint, field of view, and gaze direction of the user viewing the 3D object image may change. The projection device (100) can change the 3D effect appearing in the 3D object image according to the changed viewpoint, field of view, and gaze direction of the user. For example, the projection device (100) can adjust the position, direction, and depth information of objects included in the 3D object image based on the changed viewpoint, field of view, and gaze direction of the user.

[0072] Additionally, the projection device (100) can adjust the background image based on the screen's spatial information and the user's location information. For example, the view of the background image projected onto the screen can be adjusted based on the user's location information. If the user moves to the left, the projection device (100) can adjust the background image so that the image is added to the right of the existing background image and a portion of the left side is deleted. This will be described in more detail with reference to FIG. 9 below.

[0073] A projection device (100) according to one embodiment can synthesize an image to be projected, a 3D object image, and a background image (S350).

[0074] For example, the projection device (100) can generate a virtual space image by synthesizing a 3D object image and a background image. The projection device (100) can synthesize a 3D object image and a background image by overlapping the 3D object image on the background image.

[0075] Additionally, the projection device (100) can generate a final image by overlapping the image to be projected (image content) on the virtual space image. For example, a 3D object image may include an object (e.g., a display) for displaying the image to be projected. The projection device (100) can overlap the image to be projected (image content) on the object (e.g., the display).

[0076] A projection device (100) according to one embodiment can project a final image onto a screen or space (S360).

[0077] A projection device (100) according to one embodiment can provide users with the experience of viewing video content in various environments by projecting both video content to be viewed and virtual space images onto a screen or space. Furthermore, the projection device (100) can adjust 3D object images and background images based on the user's (viewer's) position. Accordingly, the user's sense of realism regarding the virtual space images can be enhanced.

[0078] FIG. 4 is a diagram showing an example of an operation of a projection device according to one embodiment to obtain spatial information of a screen and location information of a user.

[0079] Referring to FIG. 4, a projection device (100) according to one embodiment may include at least one sensor. The at least one sensor may include an image sensor, a depth sensor, and an infrared sensor. However, the present invention is not limited thereto, and may further include a direction sensor, an illuminance sensor, and the like.

[0080] A projection device (100) according to one embodiment can obtain spatial information of a screen using an image sensor and a depth sensor. At this time, the image sensor and the depth sensor can be placed in front of the projection device (100).

[0081] The image sensor may include at least one camera, and may acquire an image capturing a real space included in the field of view (FOV) of the camera.

[0082] The depth sensor may include at least one camera and may acquire depth information about the actual space included in the camera's FOV. For example, the depth sensor may sense the distance between the projection device (100) and the screen (410).

[0083] The projection device (100) can receive an image and depth information of a space including a screen (410) from an external device.

[0084] A projection device (100) according to one embodiment may include a spatial information acquisition module. The spatial information acquisition module may be a module that performs a function of acquiring spatial information of a screen (410). The spatial information acquisition module according to one embodiment may include appropriate logic, circuitry, interfaces, and / or codes that enable acquiring spatial information of the screen (410).

[0085] The spatial information acquisition module can acquire spatial information of the screen (410) based on at least one of an image captured by an image sensor and depth information acquired by a depth sensor.

[0086] For example, the spatial information acquisition module can recognize edges between multiple surfaces in a space composed of multiple surfaces, and, based on the recognized edges, determine the boundaries of a screen (410) on which an image will be projected. Once the boundaries of the screen (410) are determined, the spatial information acquisition module can acquire information about the shape, characteristics, size, location, etc. of the screen (410). However, the present invention is not limited thereto.

[0087] In addition, the spatial information acquisition module may acquire spatial information of the screen (410) by analyzing at least one of an image captured by an image sensor and depth information acquired from a depth sensor using, for example, at least one of a machine learning, neural network, or deep learning algorithm, as a rule-based or artificial intelligence algorithm. For example, the spatial information acquisition module may acquire spatial information of the screen (410) by recognizing an edge in a space composed of a plurality of surfaces using a neural network trained to recognize an edge from an image or depth information about the space. However, the present invention is not limited thereto.

[0088] According to one embodiment, a projection device (100) can recognize a user (420) using an image sensor or an infrared sensor. By analyzing an image captured by the image sensor, the projection device (100) can determine whether a user (420) is present in the image, and if the user (420) is present, obtain location information of the user (420).

[0089] Alternatively, the projection device (100) can use an infrared sensor to identify the presence of a user (420) and obtain the distance to the user (420).

[0090] A projection device (100) according to one embodiment may include a user recognition module. The user recognition module may be a module that performs a function of obtaining location information of a user (420). The user recognition module according to one embodiment may include appropriate logic, circuitry, interfaces, and / or code that enable obtaining location information of the user (420).

[0091] The user recognition module can obtain location information of the user (420) based on at least one of the images captured by the image sensor and the infrared sensor.

[0092] FIGS. 5 and 6 are drawings illustrating an example of a method for generating a background image and a 3D object image by a projection device according to one embodiment.

[0093] Referring to FIG. 5, a projection device (100) according to one embodiment can set a background image and a 3D object image based on user input. In addition, the projection device (100) can set the type or form of a display device (e.g., TV) to be included in a 3D object image based on user input.

[0094] A projection device (100) according to one embodiment can project a setting menu screen (510) onto a screen or space. The setting menu screen (510) according to one embodiment can include menus for setting a background image, a 3D object image, and the type or shape of a display device.

[0095] A projection device (100) according to one embodiment can receive user input for selecting menus through a control device, and based on the user input, can set the type or shape of a background image, a 3D object image, and a display device.

[0096] For example, the settings menu screen (510) may include a first menu (520) that allows selection of any one of the pre-stored candidate background images. Based on a user input, if any one of the pre-stored candidate background images is selected, the projection device (100) may generate a background image to be projected based on the selected background image.

[0097] Additionally, the setting menu screen (510) may include a second menu (530) that allows selection of any one of the pre-stored candidate template images. Based on a user input, if any one of the pre-stored candidate template images is selected, the projection device (100) may generate a 3D object image to be projected based on the selected template image.

[0098] Although not shown, the settings menu screen (510) may include a menu that allows setting the type or shape of the display object for displaying the image content to be projected. However, the present invention is not limited thereto.

[0099] Additionally, the settings menu screen (510) may include a preview showing a virtual space image (540) synthesized from the selected first background image (521) and the first template image (531) based on user input.

[0100] Although the setting menu screen (510) is described as being projected by the projection device (100) in FIG. 5, it is not limited thereto, and the setting menu screen (510) may be displayed on an external device that is communicatively connected to the projection device (100). For example, the external device that is communicatively connected to the projection device (100) may include a display, and the setting menu screen (510) of FIG. 5 may be displayed on the display of the external device. When a background image or template image is selected based on a user input, the external device may transmit information about the selected background image and template image to the projection device (100). However, the present invention is not limited thereto.

[0101] Referring to FIG. 6, a projection device (100) according to one embodiment can generate a virtual space image by synthesizing a selected background image and a template image (3D object image) based on a user input.

[0102] A projection device (100) according to one embodiment can adjust a selected background image and a template image (3D object image) based on spatial information of a screen and positional information of a user (610), and generate a virtual space image in which the adjusted background image (620) and the 3D object image (630) are synthesized.

[0103] For example, the projection device (100) can determine the size or shape of the background image (620) and the 3D object image (630) based on the size and shape of the screen and the distance to the screen.

[0104] The projection device (100) can adjust the 3D object image (630) based on the position information of the screen and the position information of the user (610). For example, if the position of the user (610) changes, the viewpoint, direction of sight, field of view, etc. of the user (610) may change. The projection device (100) can adjust the position, direction, depth information, etc. of objects included in the 3D object image (630) based on the changed viewpoint, field of view, direction of sight, etc. of the user (610).

[0105] The projection device (100) can adjust the background image (620) according to the changed viewpoint, field of view, and gaze direction of the user (610).

[0106] The projection device (100) can generate a final image by synthesizing an image (image content) to be projected onto a virtual space image that is a composite of an adjusted background image (620) and a 3D object image (630). The projection device (100) can project the final image onto a screen or space.

[0107] FIG. 7 is a drawing for explaining an example of an operation of determining a background image based on an image to be projected by a projection device according to one embodiment.

[0108] Referring to FIG. 7, a projection device (100) according to one embodiment can determine a background image based on category information of an image (image content) to be projected. The image content to be projected according to one embodiment may be a TV broadcast program. The projection device (100) can obtain EPG (Electronic Program Guide) information or TV program schedule information. The projection device (100) can obtain category information of a TV broadcast program based on the obtained EPG information or TV program schedule information. The projection device (100) can determine a background image based on the category information of the TV broadcast program.

[0109] For example, as illustrated in FIG. 7, if the video content is a news broadcast (710), the projection device (100) can determine the background video as a city view video (720).

[0110] When the video content is a nature documentary program (730), the projection device (100) can determine the background video as a forest view video (740) or an ocean view video.

[0111] If the video content is an entertainment program, the projection device (100) can determine the background video as a theme park view video.

[0112] The projection device (100) can determine a background image based on the name, title, etc. of the video content. For example, if the title of the video content includes "animal" or the video content is related to animals, the projection device (100) can determine the background image as a forest view image (740). However, the present invention is not limited thereto.

[0113] The projection device (100) can analyze the image content to be projected to obtain information about the mood or theme of the image. For example, the projection device (100) can include an image analysis module including appropriate logic, circuits, interfaces, and / or codes that operate to analyze the image and obtain information about the category, mood, theme, etc. of the image. The image analysis module can obtain information about the category, mood, theme, etc. of the image using one or more neural networks. The projection device (100) can determine a background image based on information about the mood or theme, etc. of the image.

[0114] Figure 8 is a flowchart showing an example of a method of operating a projection device according to one embodiment.

[0115] Referring to FIG. 8, a projection device (100) according to one embodiment can obtain an image to be projected (S810).

[0116] According to one embodiment, the image may be an image previously stored in the projection device (100) or an image received from an external device. However, the present invention is not limited thereto. In addition, the image according to one embodiment may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion have been performed in the video processing unit.

[0117] When the projection device (100) projects video content, the projection device (100) can sequentially acquire video frame images included in the video content.

[0118] A projection device (100) according to one embodiment can identify whether the user's position has changed (S820).

[0119] For example, the projection device (100) can recognize a user around the projection device (100) using at least one camera (e.g., an image sensor, an infrared sensor, etc.) and identify whether the user's position changes.

[0120] According to one embodiment, the projection device (100) can maintain the virtual space image used when projecting the previous frame image without changing it if there is no change in the user's position (S830).

[0121] For example, the projection device (100) can continuously maintain the projected 3D object image and background image together with the previous frame image if the user's position does not change.

[0122] On the other hand, the projection device (100) can change the 3D object image and background image when the user's position changes (S840).

[0123] For example, when the user's position changes, the user's viewpoint, field of view, and direction of gaze, etc., while viewing the 3D object image and background image projected on the screen, may change. The projection device (100) may change the 3D effect appearing in the 3D object image according to the changed viewpoint, field of view, and direction of gaze of the user. For example, the projection device (100) may adjust the position, direction, and depth information of objects included in the 3D object image based on the changed viewpoint, field of view, and direction of gaze of the user.

[0124] Additionally, the projection device (100) can adjust the background image according to the changed user's viewpoint, field of view, gaze direction, etc.

[0125] A projection device (100) according to one embodiment can synthesize an image to be projected, a 3D object image, and a background image (S850).

[0126] For example, the projection device (100) can generate a virtual space image by synthesizing a 3D object image and a background image. The projection device (100) can synthesize a 3D object image and a background image by overlapping the 3D object image on the background image.

[0127] Additionally, the projection device (100) can generate a final image by overlapping the image to be projected (image content) on a virtual space image. For example, a 3D object image may include an object (e.g., a display) for displaying the image to be projected. The projection device (100) can overlap the image to be projected (image content) on the object (e.g., the display).

[0128] A projection device (100) according to one embodiment can project a final image onto a screen or space (S860).

[0129] A projection device (100) according to one embodiment can provide users with the experience of viewing video content in various environments by projecting both video content to be viewed and virtual space images onto a screen or space. Furthermore, the projection device (100) can adjust 3D object images and background images based on the user's (viewer's) position. Accordingly, the user's sense of realism regarding the virtual space images can be enhanced.

[0130] FIG. 9 is a drawing for explaining an example of an operation of a projection device according to one embodiment to change a virtual space image as a user's position changes.

[0131] Referring to FIG. 9, a projection device (100) according to one embodiment can obtain spatial information of a screen (910) and positional information of a user (920). Based on the spatial information of the screen (910) and positional information of the user (920), the projection device (100) can generate a first three-dimensional object image (931) and a first background image (932).

[0132] The projection device (100) can generate a first virtual space image by synthesizing a first three-dimensional object image (931) and a first background image (932). The projection device (100) can overlap the first three-dimensional object image (931) onto the first background image (932) to synthesize the first three-dimensional object image (931) and the first background image (932).

[0133] The projection device (100) can generate a first final image by overlapping an image to be projected onto a first virtual space image. The projection device (100) can project the first final image onto a screen (910) or space.

[0134] According to one embodiment, the projection device (100) can detect a change in the position of the user (920). For example, the projection device (100) can detect that the position of the user (920) has changed from a first point to a second point using at least one camera.

[0135] When the position of the user (920) changes from the first point to the second point, the projection device (100) can change the first virtual space image. For example, when the user (920) moves to the left, the user's (920) viewpoint, gaze direction, field of view, etc. can change.

[0136] The projection device (100) can adjust the position, direction, depth information, etc. of objects included in the first 3D object image (931) based on the changed viewpoint, field of view, gaze direction, etc. of the user (920). For example, the projection device (100) can generate the second 3D object image (941) such that the depth of the left wall of the window included in the first 3D object image (931) decreases and the depth of the right wall of the window increases. However, the present invention is not limited thereto.

[0137] The projection device (100) can adjust the first background image (932) based on the changed viewpoint, field of view, gaze direction, etc. of the user (920). The projection device (100) can adjust the first background image (932) so that the view shown through the window changes according to the changed viewpoint, field of view, gaze direction, etc. of the user (920). For example, when the user (920) moves to the left, the projection device (100) can generate a second background image (942) in which an image is added to the right of the first background image (932) and a left portion of the first background image (932) is deleted. However, the present invention is not limited thereto.

[0138] Fig. 10 is a flowchart showing an example of a method of operating a projection device according to one embodiment.

[0139] Referring to FIG. 10, a projection device (100) according to one embodiment can obtain an image to be projected (S1010).

[0140] According to one embodiment, the image may be an image previously stored in the projection device (100) or an image received from an external device. However, the present invention is not limited thereto. In addition, the image according to one embodiment may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion have been performed in the video processing unit.

[0141] When the projection device (100) projects video content, the projection device (100) can sequentially acquire video frame images included in the video content.

[0142] A projection device (100) according to one embodiment can identify whether time information or weather information has changed (S1020).

[0143] For example, the projection device (100) can obtain time information or weather information from an external device or server. The projection device (100) can obtain time information using an internal timer. However, the present invention is not limited thereto.

[0144] According to one embodiment, the projection device (100) can maintain the virtual space image being projected without changing it if there is no change in time information or weather information (S1030).

[0145] For example, the projection device (100) can continuously maintain the projected 3D object image and background image together with the previous frame image when time information or weather information does not change.

[0146] On the other hand, the projection device (100) can change the 3D object image and background image when there is a change in time information or weather information (S1040).

[0147] For example, when time information changes from daytime to nighttime, the projection device (100) can change the background image to an image representing a night scene. Based on the time information, the projection device (100) can adjust the shadow effect appearing on the background image and the 3D object image.

[0148] The projection device (100) can change the background image based on weather information so that the current weather appears in the background image. For example, if the current weather is rainy or snowy, the projection device (100) can change the background image to a background image showing rain or snow. However, this is not limited thereto.

[0149] A projection device (100) according to one embodiment can synthesize an image to be projected, a 3D object image, and a background image (S1050).

[0150] For example, the projection device (100) can generate a virtual space image by synthesizing a 3D object image and a background image. The projection device (100) can synthesize a 3D object image and a background image by overlapping the 3D object image on the background image.

[0151] Additionally, the projection device (100) can generate a final image by overlapping the image to be projected (image content) on a virtual space image. For example, a 3D object image may include an object (e.g., a display) for displaying the image to be projected. The projection device (100) can overlap the image to be projected (image content) on the object (e.g., the display).

[0152] A projection device (100) according to one embodiment can project a final image onto a screen or space (S1060).

[0153] A projection device (100) according to one embodiment can provide users with the experience of viewing video content in various environments by projecting both video content and virtual space images onto a screen or space. Furthermore, the projection device (100) can adjust 3D object images and background images based on time information or weather information. Accordingly, the user's sense of realism regarding the virtual space images can be enhanced.

[0154] FIG. 11 is a drawing for explaining an example of an operation of a projection device according to one embodiment to change a virtual space image as time information changes.

[0155] Referring to FIG. 11, a projection device (100) according to one embodiment can obtain current time information. For example, the projection device (100) can receive current time information from an external device or external server. The projection device (100) can obtain time information using an internal timer. However, the present invention is not limited thereto.

[0156] A projection device (100) according to one embodiment can apply current time information to a background image. For example, as illustrated in FIG. 11, if the current time is daytime (e.g., 1 p.m.), the projection device (100) can set the background image to a daytime city view (1110). In addition, the projection device (100) can adjust a shadow effect appearing on the background image and a 3D object image based on the position of the sun at the corresponding time and the projection direction according to the current time information.

[0157] On the other hand, if the current time is nighttime (e.g., 10 p.m.), the projection device (100) can change the background image to a nighttime city view (1120). The projection device (100) can adjust the color and brightness of the 3D object image based on the current time information. However, the present invention is not limited thereto.

[0158] A projection device (100) according to one embodiment can improve the sense of reality felt by a user for a virtual space image by adjusting a 3D object image and a background image according to current time information.

[0159] FIGS. 12 and 13 are drawings showing examples of a projection device according to one embodiment providing various virtual space images.

[0160] Referring to FIG. 12, a projection device (100) according to one embodiment can project a virtual space image (1220) including an object image including a cave and a background image representing a coastal view.

[0161] The projection device (100) can generate a virtual space image (1220) based on the spatial information of the screen and the location information of the user (1210), so that the user (1210) can feel as if he or she is located inside a cave and looking out of the cave from the user's (1210) perspective.

[0162] For example, the projection device (100) may adjust the brightness of the image representing the cave so that the user (1210) feels as if he or she is moving toward the outside of the cave as he or she moves closer to the screen. The projection device (100) may also adjust the coastal view so that the parts of the background image that were not displayed are displayed more as the user (1210) moves closer to the screen. However, the present invention is not limited thereto.

[0163] In addition, referring to FIG. 13, the projection device (100) according to one embodiment can project a virtual space image (1320) that exhibits a dimension change effect. For example, the projection device (100) can generate a virtual space image (1320) that includes an object image (e.g., a wormhole) representing a passageway connecting to another space-time, as illustrated in FIG. 13, and a background image that appears within the passageway. The projection device (100) can project image content within the object (1330) representing the passageway. However, the present invention is not limited thereto.

[0164] A projection device (100) according to one embodiment can provide a user with a variety of viewing experiences by generating various virtual space images and projecting them together with image content.

[0165] FIG. 14 is a drawing for explaining an example of the operation of a projection device according to one embodiment when multiple users are recognized.

[0166] According to one embodiment, a projection device (100) can recognize a first user (1410) among a plurality of users. The projection device (100) can generate a three-dimensional object image and a background image based on the position information of the recognized first user (1410) and the spatial information of the screen. The projection device (100) can generate a virtual space image by synthesizing the three-dimensional object image and the background image, and can generate a final image by synthesizing an image to be projected onto the virtual space image. The projection device (100) can project the final image onto the screen.

[0167] The projection device (100) can additionally recognize a second user (1420) among multiple users while projecting the final image based on the location of the first user (1410).

[0168] The projection device (100) can project a notification message (1430) asking whether to terminate virtual space image projection on the screen or space when a second user (1420) is additionally recognized. When terminating virtual space image projection, the projection device (100) can project only the image (image content) to be projected on the screen or space.

[0169] If the distance between the point where the second user (1420) is located and the point where the first user (1410) is located is less than a preset value, the projection device (100) can continue to project the final image without projecting a notification message (1430).

[0170] The projection device (100) may project a notification message on a screen or in space to move the location of the second user (1420) closer to the first user (1410), but is not limited thereto.

[0171] FIG. 15 is a drawing for explaining an example of an operation of a projection device according to one embodiment to project a virtual space image.

[0172] Referring to FIG. 15, a projection device (100) according to one embodiment may include a display device (1510) on a screen or space to be projected. The display device (1510) according to one embodiment may be implemented in various forms, such as a TV including a display, a smart monitor, a video wall, a digital signage, etc. In addition, the display device (1510) may be a fixed electronic device placed in a fixed location. However, the present invention is not limited thereto.

[0173] The projection device (100) can project a virtual space image generated using the method described in FIGS. 1 to 14 onto a screen or space. At this time, the 3D object image included in the virtual space image may not include a display object. The image content can be displayed via a display device (1510).

[0174] A projection device (100) according to one embodiment can recognize the position, size, shape, etc. of a display device (1510) and generate a virtual space image based on the recognized position, size, shape, etc. of the display device (1510). However, the present invention is not limited thereto.

[0175] According to one embodiment, a projection device (100) and a display device (1510) may be connected to each other through communication. The display device (1510) may transmit information about displayed image content to the projection device (100). The projection device (100) may generate a virtual space image based on the information about the image content. For example, the projection device (100) may determine a 3D object image or a background image based on information about the category information, atmosphere, theme, etc. of the image content. However, the present invention is not limited thereto.

[0176] Although the operations of the projection device (100) according to one embodiment are described in FIGS. 1 to 15 , the operations described above are not limited to the operations of the projection device (100). For example, the operations of the projection device (100) according to one embodiment described in FIGS. 1 to 15 may be performed by a display device including a display. The display device may include a large display. The display device may generate a virtual space image including a 3D object image and a background image described in FIGS. 1 to 15 . In addition, the display device may synthesize image content into the virtual space image and display it on the display.

[0177] For example, a display device can obtain user location information using at least one camera. The display device can generate or adjust a background image and a 3D object image based on the size of the display and the user's location information. The display device can change a 3D effect appearing in the 3D object image when the user's location changes. The display device can adjust the view of the background image displayed on the display based on the user's location information. Alternatively, the display device can change or adjust the 3D object image and background image based on time information or weather information.

[0178] The display device can perform the above-described operations by applying the operations of the projection device (100) described in FIGS. 1 to 15. In addition to the above-described operations, the operations of the projection device (100) described in FIGS. 1 to 15 can also be performed by the display device.

[0179] Display devices can display not only the video content to be viewed but also virtual space images, providing users with the experience of viewing video content in a variety of environments. Furthermore, display devices can adjust 3D object and background images based on the user's (viewer's) position, thereby enhancing the user's sense of realism in virtual space images.

[0180] Fig. 16 is a block diagram showing an example of the configuration of a projection device according to one embodiment.

[0181] Referring to FIG. 16, a projection device (100) according to one embodiment may include a sensor unit (e.g., including at least one sensor) (110), a processor (e.g., including at least one processing circuit) (120), a memory (130), a projection unit (e.g., including at least one projection circuit) (140), and a communication unit (e.g., including at least one communication circuit) (150).

[0182] The sensor unit (110) according to one embodiment includes at least one sensor, detects a state around the projection device (100), and can transmit the detected information to the processor (120). The sensor unit (110) may include, for example, but is not limited to, an image sensor, a depth sensor, an infrared sensor, etc.

[0183] An image sensor according to one embodiment can acquire image frames, such as still images or video images. For example, the image sensor can capture images outside the projection device (100). Images captured by the image sensor can be processed by the processor (120) or a separate image processor.

[0184] A depth sensor according to one embodiment can obtain depth information about one or more objects included in a space. The depth information can correspond to a distance from the depth sensor to a specific object, and the greater the distance from the depth sensor to the specific object, the greater the depth value. A depth sensor according to one embodiment can obtain depth information about an object in various ways, and for example, can obtain depth information using at least one of a time-of-flight (TOF) method, a stereo image method, and a structured light method.

[0185] A depth sensor according to one embodiment may include at least one camera, and may acquire depth information about a real space included in a FOV of a camera included in the depth sensor.

[0186] In addition, the sensor unit (110) may include, for example, but not limited to, an image sensor, a depth sensor, an infrared sensor, an acceleration sensor, a position sensor, a temperature / humidity sensor, an illuminance sensor, a geomagnetic sensor, a gyroscope sensor, and a microphone, in addition to an image sensor, a depth sensor, and an infrared sensor. However, the present invention is not limited thereto.

[0187] A communication unit (150) according to one embodiment includes various communication circuits and can transmit and receive data or signals with an external device or server. For example, the communication unit (150) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0188] The Wi-Fi module and Bluetooth module perform communication in the Wi-Fi and Bluetooth modes, respectively. When using the Wi-Fi module or Bluetooth module, various connection information such as the SSID and session key are first transmitted and received, and after establishing a communication connection using this, various information can be transmitted and received. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0189] According to one embodiment, the communication unit (150) can receive an image or image content to be projected from an external device. In addition, according to one embodiment, the communication unit (150) can receive time information, weather information, etc. from an external device or an external server.

[0190] Additionally, the communication unit (150) according to one embodiment can communicate with the control device. The communication unit (150) can receive a signal corresponding to a user input from the control device.

[0191] A processor (120) according to one embodiment includes various processing circuits and performs functions of controlling the overall operation of the projection device (100) and signal flow between internal components of the projection device (100), and processing data.

[0192] The processor (120) may include single cores, dual cores, triple cores, quad cores, and multiples thereof. Furthermore, the processor (120) may include multiple processors. For example, the processor (120) may be implemented as a main processor (not shown) and a subprocessor (not shown).

[0193] The processor (120) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). According to an embodiment, the processor (120) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (120) may further include an NPU (Neural Processing Unit). The processor (120) according to an embodiment of the present disclosure may include various processing circuits and / or a plurality of processors. For example, the term “processor” as used in this document, including the claims, may include various processing circuits including at least one processor, and one or more of the at least one processor may be configured to individually and / or collectively perform various functions described herein in a distributed manner. As used herein, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms are not limited to situations where, for example, one processor performs some of the recited functions and other processor(s) perform other of the recited functions, but can encompass situations where a single processor can perform all of the recited functions. Additionally, the at least one processor can comprise a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. The at least one processor can execute program instructions to accomplish or perform the various functions.

[0194] According to one embodiment, the memory (130) can store various data, programs or applications for driving and controlling the projection device (100).

[0195] Additionally, the program stored in the memory (130) may include one or more instructions. The program (one or more instructions) or application stored in the memory (130) may be executed by the processor (120).

[0196] According to one embodiment, the processor (120) can obtain an image to be projected by executing one or more instructions stored in the memory (130). The image according to one embodiment may be an image previously stored in the memory (130) or an image received from an external device via the communication unit (150). In addition, the image may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion are performed in the video processing unit.

[0197] According to one embodiment, the processor (120) may acquire an image (image content) to be projected by executing one or more instructions stored in the memory (130). The image according to one embodiment may be an image previously stored in the memory (130) or an image received from an external device via the communication unit (150). However, the present invention is not limited thereto. In addition, the image according to one embodiment may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion are performed in the video processing unit.

[0198] According to one embodiment, the processor (120) can acquire spatial information of the screen by executing one or more instructions stored in the memory (130). For example, the processor (120) can acquire an image of the screen captured by an image sensor included in the sensor unit (110), and can acquire depth information of the screen from a depth sensor included in the sensor unit (110). However, the present invention is not limited thereto.

[0199] The processor (120) can receive an image captured by the screen and depth information of the screen from an external device through the communication unit (150).

[0200] The processor (120) can obtain spatial information of the screen based on the captured image and depth information. The spatial information of the screen may include the shape of the screen, distance information from the projection device (100) to the screen, position information of the screen, size information, position information for the corners (boundaries) of the screen, etc.

[0201] According to one embodiment, a processor (120) can obtain user location information by executing one or more instructions stored in a memory (130).

[0202] For example, the processor (120) can obtain user location information using an image sensor or infrared sensor included in the sensor unit (110). The processor (120) can identify whether the user is present, the distance between the user and the projection device (100), the direction in which the user is located relative to the projection device (100), etc. However, the present invention is not limited thereto.

[0203] According to one embodiment, a processor (120) can generate a 3D object image and a background image based on spatial information of the screen and location information of the user by executing one or more instructions stored in a memory (130).

[0204] The processor (120) can set a 3D object image and a background image based on a user input. For example, the processor (120) can set a 3D object image based on a user input for selecting one of candidate template images pre-stored in the memory (130). The memory (130) can pre-store candidate template images having different shapes, sizes, positions, etc. of windows included in the image. The processor (120) can set a background image based on a user input for selecting one of candidate background images pre-stored in the memory (130). The memory (130) can pre-store candidate background images such as an ocean, a beach, a mountain, and a city.

[0205] The processor (120) can set a background image based on the category information of the image to be projected.

[0206] According to one embodiment, the processor (120) executes one or more instructions stored in the memory (130), and when the 3D object image and the background image are set, the processor (120) can adjust the 3D object image and the background image based on the spatial information of the screen and the user's location information.

[0207] For example, the processor (120) may determine the sizes of the 3D object image and the background image based on the spatial information of the screen. The processor (120) may adjust the sizes of the 3D object image and the background image so that the 3D object image and the background image are projected onto the entire wall based on the recognized corners of the wall. However, the present invention is not limited thereto.

[0208] The processor (120) can adjust the depth information of objects included in the 3D object image based on the spatial information of the screen and the user's position information. For example, when the user's position changes, the user's viewpoint, field of view, and gaze direction while viewing the 3D object image may change. The processor (120) can change the 3D effect appearing in the 3D object image according to the changed user's viewpoint, field of view, and gaze direction. For example, the processor (120) can adjust the position, direction, and depth information of objects included in the 3D object image based on the changed user's viewpoint, field of view, and gaze direction.

[0209] The processor (120) can adjust the background image based on the screen's spatial information and the user's location information. For example, the view of the background image projected onto the screen can be adjusted based on the user's location information. If the user moves to the left, the processor (120) can adjust the background image so that an image is added to the right of the existing background image and a portion of the left side is deleted.

[0210] According to one embodiment, a processor (120) can synthesize an image to be projected, a 3D object image, and a background image by executing one or more instructions stored in a memory (130).

[0211] For example, the processor (120) can generate a virtual space image by synthesizing a 3D object image and a background image. The processor (120) can synthesize the 3D object image and the background image by overlapping the 3D object image on the background image.

[0212] The processor (120) can generate a final image by overlapping the image (image content) to be projected onto a virtual space image. For example, a 3D object image may include an object (e.g., a display) for displaying the image to be projected. The processor (120) can overlap the image (image content) to be projected onto the object (e.g., the display).

[0213] The processor (120) can control the projection unit (140) to project a final image synthesized from the background image and the image to be projected onto a screen or space.

[0214] The projection unit (140) according to one embodiment may include, for example, but is not limited to, various projection circuits including a light source that generates light, a lens, etc., and may further include a driving unit that adjusts the direction, position, angle, etc. of projection. The projection unit (140) may perform projection of a final image by driving a light source or adjusting the direction, position, angle, etc. of projection according to a control signal received from the processor (120).

[0215] The block diagram of the projection device (100) illustrated in FIG. 16 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the projection device (100) actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for explaining embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0216] When the projection device (100) according to one embodiment is implemented as a display device, it may further include a display. The display may display a final image including a virtual space image and image content generated by the processor (120).

[0217] A projection device according to one embodiment may include a projection unit including a projection circuit, a memory storing one or more instructions, and a processing circuit, and may include at least one processor executing the one or more instructions.

[0218] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to obtain a first image.

[0219] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to obtain spatial information of the screen and positional information of the user.

[0220] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to generate a three-dimensional object image and a background image based on spatial information of the screen and positional information of the user.

[0221] The one or more instructions, when individually and / or collectively executed by the at least one processor, can control the projection unit to project a second image that is a composite of the first image, the three-dimensional object image, and the background image.

[0222] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to set the three-dimensional object image and the background image based on a user input.

[0223] The one or more instructions, when individually and / or collectively executed by the at least one processor, can cause the projection device to adjust the set 3D object image and the background image based on the user's position information and the spatial information of the screen.

[0224] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to obtain category information of the first image.

[0225] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to determine the background image based on the category information.

[0226] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to determine depth information for at least one object included in the 3D object image based on positional information of the user and spatial information of the screen.

[0227] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to adjust at least one of the three-dimensional object image and the background image according to a direction of the user's gaze based on a change in the user's position information.

[0228] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to obtain at least one of time information and weather information.

[0229] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to adjust at least one of the three-dimensional object image and the background image based on at least one of the time information and the weather information.

[0230] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to obtain a projection direction.

[0231] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to adjust a shadow effect appearing in at least one of the three-dimensional object image and the background image based on the time information and the projection direction.

[0232] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to composite the first image onto the at least one object based on a shape and size of the at least one object included in the three-dimensional object image.

[0233] The one or more instructions, when individually and / or collectively executed by the at least one processor, may cause the projection device to determine at least one of a projection direction, a projection position, and a projection size of the second image based on spatial information of the screen.

[0234] The above projection device may further include at least one camera.

[0235] The one or more instructions, when individually and / or collectively executed by the at least one processor, enable the projection device to obtain spatial information of the screen and location information of the user based on an image captured by the at least one camera.

[0236] A method of operating a projection device according to one embodiment may include a step of acquiring a first image.

[0237] A method of operating a projection device according to one embodiment may include a step of obtaining spatial information of a screen.

[0238] A method of operating a projection device according to one embodiment may include a step of obtaining location information of a user.

[0239] A method of operating a projection device according to one embodiment may include a step of generating a three-dimensional object image and a background image based on spatial information of the screen and positional information of the user.

[0240] A method of operating a projection device according to one embodiment may include a step of projecting a second image synthesized from the first image, the three-dimensional object image, and the background image.

[0241] A method of operating a projection device according to one embodiment may further include a step of setting the three-dimensional object image and the background image based on a user input.

[0242] The step of generating the 3D object image and background image may include a step of adjusting the set 3D object image and background image based on the user's location information and the spatial information of the screen.

[0243] A method of operating a projection device according to one embodiment may further include a step of obtaining category information of the first image.

[0244] A method of operating a projection device according to one embodiment may further include a step of determining the background image based on the category information.

[0245] The step of generating the 3D object image and background image may include a step of determining depth information for at least one object included in the 3D object image based on the user's location information and the screen's spatial information.

[0246] The step of generating the 3D object image and background image may include a step of adjusting at least one of the 3D object image and the background image according to the user's gaze direction based on a change in the user's location information.

[0247] The above method of operation may include a step of obtaining at least one of time information and weather information.

[0248] The above operating method may further include a step of adjusting at least one of the 3D object image and the background image based on at least one of the time information and the weather information.

[0249] The above method may further include a step of obtaining a projection direction.

[0250] The step of adjusting at least one of the three-dimensional object image and the background image based on at least one of the time information and the weather information may include the step of adjusting a shadow effect appearing on at least one of the three-dimensional object image and the background image based on the time information and the projection direction.

[0251] The above operating method may further include a step of synthesizing the first image onto at least one object based on a shape and size of at least one object included in the three-dimensional object image.

[0252] The above operating method may further include a step of determining at least one of a projection direction, a projection position, and a projection size of the second image based on spatial information of the screen.

[0253] A projection device according to one embodiment can provide a user with a variety of viewing experiences by generating various virtual space images and projecting them together with image content.

[0254] The projection device can improve the user's sense of reality in a virtual space image by adjusting the 3D object image and background image according to the user's position.

[0255] In addition, a projection device according to one embodiment can improve the sense of reality felt by a user for a virtual space image by adjusting a 3D object image and a background image according to time information or weather information.

[0256] The method of operating a projection device according to one embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a non-transitory computer-readable medium. The non-transitory computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known and available to those skilled in the art of computer software. Examples of the non-transitory computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0257] Additionally, the method of operating a projection device according to the disclosed embodiments may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer.

[0258] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by an electronic device manufacturer or through an electronic marketplace (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic marketplace server, or a relay server that temporarily stores the software program.

[0259] In a system comprising a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0260] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0261] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server, thereby controlling a client device in communication with the server to perform a method according to the disclosed embodiments.

[0262] While the present disclosure has been illustrated and described with reference to various exemplary embodiments, it will be understood that the various exemplary embodiments are illustrative and not limiting. Those skilled in the art will also appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure, including the appended claims. It will also be appreciated that any embodiment(s) described herein may be used in conjunction with any other embodiment(s) described herein.

Claims

1. In the projection device (100), A projection unit (140) including a projection circuit; A memory (130) storing one or more instructions; and A processing circuit comprising at least one processor (120) for executing one or more instructions, The one or more instructions, when individually and / or collectively executed by the at least one processor, cause the projection device to Obtain the first image, Obtain spatial information of the screen and location information of the user, Based on the spatial information of the above screen and the location information of the user, a 3D object image and a background image are generated, A projection device that controls the projection unit to project a second image that is a composite of the first image, the three-dimensional object image, and the background image.

2. In paragraph 1, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to Based on user input, the 3D object image and the background image are set, A projection device that adjusts the set three-dimensional object image and the background image based on the location information of the user and the spatial information of the screen.

3. In paragraph 1, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to Obtain category information of the first image above, A projection device that determines the background image based on the above category information.

4. In any one of paragraphs 1 to 3, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to A projection device that determines depth information for at least one object included in the three-dimensional object image based on the location information of the user and the spatial information of the screen.

5. In any one of paragraphs 1 to 4, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to A projection device that adjusts at least one of the three-dimensional object image and the background image according to the direction of the user's gaze based on a change in the user's location information.

6. In any one of paragraphs 1 to 5, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to Obtain at least one of time information and weather information, A projection device that adjusts at least one of the three-dimensional object image and the background image based on at least one of the time information and the weather information.

7. In paragraph 6, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to Obtain the projection direction, A projection device that adjusts a shadow effect appearing in at least one of the three-dimensional object image and the background image based on the time information and the projection direction.

8. In any one of paragraphs 1 to 7, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to A projection device that synthesizes the first image onto at least one object based on the shape and size of at least one object included in the three-dimensional object image.

9. In any one of paragraphs 1 to 8, The above one or more instructions, when individually and / or collectively executed by the at least one processor (120), cause the projection device to A projection device that determines at least one of a projection direction, a projection position, and a projection size of the second image based on spatial information of the screen.

10. In any one of paragraphs 1 to 9, The above projection device, Including at least one additional camera, A projection device that obtains spatial information of the screen and location information of the user based on images captured from at least one camera.

11. In the method of operating the projection device, Step of acquiring the first image (S310); Step of acquiring spatial information of the screen (S320); Step of obtaining user location information (S330); A step (S340) of generating a 3D object image and a background image based on the spatial information of the above screen and the location information of the user; and A method of operating a projection device, comprising: a step (S360) of projecting a second image that is a composite of the first image, the three-dimensional object image, and the background image.

12. In paragraph 11, The above method of operation is, Further comprising a step of setting the 3D object image and the background image based on user input, The step of generating the above 3D object image and background image is: A method of operating a projection device, comprising the step of adjusting the set three-dimensional object image and the background image based on the location information of the user and the spatial information of the screen.

13. In paragraph 11, The above method of operation is, A step of obtaining category information of the first image; and A method of operating a projection device, further comprising the step of determining the background image based on the above category information.

14. In any one of paragraphs 11 to 13, The step of generating the above 3D object image and background image is: A method of operating a projection device, comprising the step of determining depth information for at least one object included in the three-dimensional object image based on the location information of the user and the spatial information of the screen.

15. A non-transitory computer-readable storage medium having stored thereon a program for performing any one of the methods of clauses 11 to 14.

Images