Electronic device and image mapping method

Abstract

An electronic device according to an aspect of the present disclosure may be configured to: acquire a raster image including one or more real objects using one or more cameras; generating a vector image by vectorizing the raster image; generating an editing screen by superimposing the vector image with the raster image; displaying, on a display, a user interface representation including an edit screen; generating a projection screen in response to one or more inputs to the user interface representation; and transmitting the projection screen to an external electronic device connected to the electronic device.

Description

Technical Field

[0001] This disclosure relates to an electronic device and an image mapping method. Background Technology

[0002] Projectors have evolved into devices with high resolution, high brightness, compact size, and / or a wide range of functions. Projectors come in various types, such as laser projectors, light-emitting diode (LED) projectors, and liquid crystal display (LCD) projectors. Furthermore, projectors can be integrated with augmented reality (AR) technology. Projectors can achieve environmentally adaptable projection results by mapping real-world objects within the projection area to virtual objects and by simply projecting images. Summary of the Invention

[0003] Technical solution Embodiments of this disclosure provide an electronic device and an image mapping method that can automatically specify regions (mapping areas) of an image to be mapped from one or more objects in a real environment.

[0004] Embodiments of this disclosure provide an electronic device and an image mapping method capable of dividing mapping regions based on each layer, combining images mapped to the divided mapping regions into a single image, and projecting the single image.

[0005] An electronic device according to an exemplary embodiment of the present disclosure may include: a display; a transceiver; one or more memories; and at least one processor, including processing circuitry, the at least one processor being electrically connected to the display, the transceiver, and the one or more memories. The at least one processor may be configured, individually and / or collectively, to: acquire captured images of one or more real objects using one or more cameras; identify a projection region included as at least a portion of the captured images based on feature information included in the captured images; acquire a first image corresponding to the projection region from the captured images; generate a second image by vectorizing the first image; generate an editing screen by overlaying the second image with the first image; control the display to display a user interface representation including the editing screen; generate a projection screen in response to one or more inputs to the user interface representation; and control the transceiver to transmit the projection screen to an external electronic device connected to the electronic device.

[0006] According to an example embodiment, the first image may include a raster image comprising one or more captured objects corresponding to the one or more real objects, and the second image may include a vector image comprising one or more vector objects corresponding to the one or more captured objects.

[0007] According to an example embodiment, the one or more vector objects may include one or more vector lines or one or more vector planes.

[0008] According to an example embodiment, the at least one processor may be configured, individually and / or collectively, to process the one or more vector objects into one or more content mapping regions, and, in response to one or more inputs, to generate one or more content layers having content mapped to the one or more content mapping regions. The projection screen may include the generated one or more content layers.

[0009] According to an example embodiment, the projection screen can be generated by overlaying one or more content layers.

[0010] According to an example embodiment, the user interface representation may include a drawing area and a content selection area. The editing screen may be displayed in the drawing area, and content options that can be mapped to the content mapping area are displayed in the content selection area.

[0011] According to an example embodiment, the at least one processor may be configured individually and / or collectively to generate a content layer having the content mapped to the content mapping region in response to receiving an input for mapping content associated with at least one of the content options in a content mapping region.

[0012] According to an example embodiment, the at least one processor may be configured, individually and / or collectively, to generate the content layer whenever an input is received for mapping content associated with at least one of the content options in the content mapping region.

[0013] According to an example embodiment, the at least one processor may be configured, individually and / or collectively, to generate a new projection screen by adding the generated content layer to a pre-generated projection screen, and to control the transceiver to send the new projection screen to the external electronic device in response to the generation of the new projection screen.

[0014] According to an example embodiment, the at least one processor may be configured, individually and / or collectively, to control the transceiver to transmit messages for controlling the external electronic device to output a standby screen. The first image may include a raster image comprising captured images of the one or more real objects projected onto the standby screen.

[0015] According to an example embodiment, the standby screen may include a striped pattern screen.

[0016] According to an example embodiment, the at least one processor may be configured, separately and / or jointly, to: acquire three-dimensional (3D) depth information based on a stripe pattern represented in the captured image, detect a 3D vector object based on the 3D depth information, and process the 3D vector object into a content-mapped region.

[0017] According to an example embodiment, the entire frame of the projection screen can be configured by synthesizing the individual frames of the one or more content layers of the projection screen. The entire frame of the projection screen can be reproduced based on the individual frames of the one or more content layers.

[0018] An electronic device according to an exemplary embodiment of the present disclosure may include: a display; one or more image projectors; one or more memories; and at least one processor, including processing circuitry, the at least one processor being electrically connected to the display, the one or more image projectors, and the one or more memories. The at least one processor may be configured, individually and / or collectively, to: acquire captured images of one or more real objects using the one or more cameras; identify a projection area included as at least a portion of the captured images based on feature information included in the captured images; acquire a first image corresponding to the projection area from the captured images; generate a second image by vectorizing the first image; generate an editing screen by overlaying the second image with the first image; control the display to display a user interface representation including the editing screen; generate a projection screen in response to one or more inputs to the user interface representation; and project the projection screen using the one or more image projectors.

[0019] An image mapping method performed by an electronic device according to an example embodiment of the present disclosure may include: acquiring captured images of one or more real objects using one or more cameras; identifying a projection region included as at least a portion of the captured images based on feature information included in the captured images; acquiring a first image corresponding to the projection region from the captured images; generating a second image by vectorizing the first image; generating an editing screen by overlaying the second image with the first image; displaying a user interface representation including the editing screen on the display; generating a projection screen in response to one or more inputs to the user interface representation; and sending the projection screen to an external electronic device connected to the electronic device. Attached Figure Description

[0020] Figure 1 These are diagrams illustrating example systems according to various embodiments; Figure 2 This is a flowchart illustrating example image mapping methods according to various embodiments; Figure 3 This is a flowchart illustrating example image mapping methods according to various embodiments; Figure 4 This is a signal flow diagram illustrating example image mapping methods according to various embodiments; Figure 5 This is a signal flow diagram illustrating example image mapping methods according to various embodiments of the present disclosure; Figure 6 This is a diagram illustrating example user interface representations according to various embodiments; Figure 7 This is a diagram illustrating example sub-user interface representations associated with a capture button according to various embodiments; Figure 8 This is a diagram illustrating example user interface representations associated with free-drawn buttons according to various embodiments; Figure 9 This is a diagram illustrating example user interface representations associated with a pen button according to various embodiments; Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 14 , Figure 15 and Figure 16 This is a diagram illustrating changes in the user interface representation according to various embodiments of the image mapping method; Figure 17 This is an exploded view showing the configuration of the projection screen according to various embodiments; Figure 18 This illustrates various embodiments. Figure 17 A diagram showing the projection result on the projection screen; Figure 19 This is a diagram illustrating frame composition of a projection screen according to various embodiments; Figure 20 This is a block diagram illustrating example configurations of user equipment according to various embodiments; and Figure 21 This is a block diagram illustrating example configurations of a projector according to various embodiments.

[0021] The following description may be referenced to the accompanying drawings, and variant examples that can be practiced are shown as examples in the drawings. Other examples may be utilized and structural changes may be made without departing from the scope of this disclosure. Detailed Implementation

[0022] Various exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, the present disclosure may be implemented in various other forms and is not limited to the embodiments set forth herein. Throughout the present disclosure and the accompanying drawings, the same or similar reference numerals may be used to refer to the same or similar elements. Furthermore, for clarity and brevity, well-known functions and configurations may not be described in the drawings and related descriptions.

[0023] Figure 1 This is a diagram illustrating example systems according to various embodiments.

[0024] Figure 1 A system for controlling a projector PJT using a user equipment (UE) is illustrated. According to embodiments of this disclosure, electronic devices such as a projector PJT and a user equipment (UE) are provided. The user equipment (UE) may include various electronic devices such as smartphones, laptops, desktop computers, or wearable devices.

[0025] Electronic devices (e.g., projector PJTs and user equipment UEs) according to embodiments of the present disclosure can communicate with each other wired or wirelessly. The projector PJT and user equipment UE may include one or more memories and at least one processor (including processing circuitry) electrically connected to one or more memories, and the methods and operations included in the methods according to various embodiments of the present disclosure can be performed by at least one processor. Methods according to embodiments of the present disclosure can be performed by at least one of the projector PJTs or user equipment UEs. At least one processor according to embodiments of the present disclosure may include various processing circuitry and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of the at least one processor may be configured individually and / or in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, cases where one processor performs some of the described functions while other processors perform other functions, and cases where a single processor can perform all the described functions. Additionally, at least one processor may include, for example, a combination of processors that perform various described / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions. In embodiments of this disclosure, the projector PJT and / or user equipment UE may use a camera CAM to capture various real objects in a real environment and the surrounding environment of those real objects. Thus, an image including various real objects and their surrounding environment can be referred to as a captured image. The captured image generated or acquired by the projector PJT and / or user equipment UE may be generated in the form of a raster image. Figure 1The user equipment (UE) shown includes a camera (CAM), but various embodiments of this disclosure are not limited thereto, and a projector (PJT) may include a camera (CAM).

[0026] The projector PJT and / or user equipment UE can generate a projection screen using a raster image acquired by a camera CAM. In one embodiment, the projection screen generated by the user equipment UE is provided to the projector PJT via a wired or wireless connection, and the projector PJT outputs the projection screen provided by the user equipment UE through an image projection unit OP. In another embodiment, the projection screen generated by the projector PJT is output by the projector PJT.

[0027] The captured image acquired by the camera (CAM) may include a projection area R1 projected onto a projection screen by the image projection unit (OP), or a projection area R1 and an outer region R2. The outer region R2 of the projection area R1 can be removed from the captured image through post-processing.

[0028] Figure 2 This is a flowchart illustrating example image mapping methods according to various embodiments.

[0029] Figure 2 Each operation can be performed by one or more electronic devices. These electronic devices can be user equipment (UE) or projector unit (PJT). Each of the following operations can be performed by one or more processing circuitry units, which can be included within the user equipment (UE) or projector unit (PJT).

[0030] In operation 210, the electronic device may use one or more cameras to acquire raster images.

[0031] An electronic device may use one or more cameras to acquire raster images of a real environment. Various objects may be included in the real environment. These objects may include, for example, sculptures, people, trees, walls, buildings, or vehicles, but are not limited to these. In this disclosure, objects existing in the real environment are referred to as real objects, and real objects captured by the electronic device are referred to as captured objects. Captured objects may be included in the raster images.

[0032] In this disclosure, the raster image initially generated or acquired by the camera can be referred to as a captured image. In other words, the raster image acquired by one or more cameras can be a captured image of the real environment in which the standby screen is projected. The electronic device can project the standby screen in one direction using an image projection unit. The standby screen can be a white screen or a striped pattern screen. When a white screen is projected, two-dimensional (2D) vector objects can be detected in subsequent operations, and when a striped pattern screen is projected, three-dimensional (3D) vector objects can be detected in subsequent operations.

[0033] If a striped pattern screen is projected onto a real 3D object, a different stripe pattern is represented on each plane of the 3D object. Electronic devices can use these differently represented stripe patterns to acquire 3D depth information. This depth information can then be used in subsequent operations via vectorization by a vectorization engine.

[0034] In operation 220, the electronic device can generate a vector image by vectorizing the raster image.

[0035] Electronic devices may include a vectorization engine as at least a part of their operation. A vectorization engine is a computational process that converts raster images into vector images that include lines, curves, and other geometric shapes. Electronic devices can use a vectorization engine to convert raster images into vector images, thereby enlarging or reducing images without loss of resolution.

[0036] The vectorization engine may be stored in at least one of the processor and / or memory of the electronic device, or configured as a circuit unit separate from or electrically connected to the processing circuit unit. Furthermore, the vectorization engine may not be included in the electronic device, but may be included in an external server capable of communicating with the electronic device. The vectorization engine included in the external server performs vectorization operations according to the request of the electronic device and responds to the electronic device with a vectorized image as a result of the vectorization.

[0037] The raster image vectorized by the vectorization engine can be a captured image or at least a portion of a captured image acquired by one or more cameras. In an embodiment, the electronic device can generate a vector image by vectorizing a raster image acquired by one or more cameras. In an embodiment, the electronic device can specify at least a portion of a captured image and generate a vector image for at least a portion of the captured image by vectorizing the specified at least a portion of the captured image.

[0038] At least a portion of the captured image to be vectorized can be specified based on at least one of feature information included in the captured image and / or user input. Feature information includes at least one of brightness differences, visible markings, invisible markings, or patterns (e.g., stripe patterns) identified in at least a portion of the captured image. Here, brightness differences, visible markings, invisible markings, or patterns can be output to the real environment via a standby screen used in embodiments of this disclosure. User input is user input for specifying at least a portion of the captured image and includes user input for adjusting the position of edit points. See below for reference. Figure 12 and Figure 13 Describe the edit points in more detail.

[0039] In this disclosure, at least a portion of the captured image to be vectorized may be referred to as an "intermediate image" as the basis for vectorization. The intermediate image may correspond to a projection area included as at least a portion of the captured image. For example, the intermediate image may be an image of the projection area included as at least a portion of the captured image. An electronic device can generate an intermediate image from a captured image by excluding portions relating to the remaining surrounding environment, other than those relating to the projection area.

[0040] Intermediate images can include primary intermediate images and secondary intermediate images. A primary intermediate image can be generated based on feature information included in the captured image. A secondary intermediate image can be generated based on user input to the primary intermediate image. For example, the primary intermediate image can be automatically generated by an electronic device based on feature information included in the captured image. In this case, the primary intermediate image may also include areas other than the projection area, or may not include at least a portion of the projection area, and the electronic device can generate the secondary intermediate image based on user input to the primary intermediate image (e.g., user input for adjusting the position of edit points).

[0041] In the following description, the raster image that can be used as the basis for generating vector images can be a captured image acquired by one or more cameras, or an intermediate image selected or generated from captured images, but for ease of description, the term "raster image" will be used.

[0042] A vectorization engine can be executed for one or more captured objects included in a raster image. The vectorization engine can convert captured objects into vector objects. The vectorization engine can generate a vector image from a raster image by converting captured objects into vector objects.

[0043] A vector object may include lines that form its external appearance and planes that form its internal appearance. For ease of description, the lines and planes of a vector object may be referred to as vector lines and vector planes, respectively. In other words, a vector object may also include one or more vector lines and one or more vector planes.

[0044] A vector image comprises one or more vector objects. Vector lines are positioned along the edges of a vector plane and can include the external appearance of the vector objects. The vector lines are positioned at the edges of the vector plane and can be configured to surround the outer perimeter of the vector plane. The enclosed region surrounded by the vector lines can be designated as the vector plane.

[0045] Each of the vector lines and vector planes is treated as an independent entity, and in another operation according to an embodiment of this disclosure, different contents can be mapped to the vector lines and vector planes.

[0046] One or more vector planes and one or more vector lines, each associated with a captured object included in the raster image, can be generated accordingly. The vector planes and vector lines can be correlated with each other because the vector planes are surrounded by the vector lines.

[0047] One or more vector lines and / or vector planes may correspond to one or more capture objects included in a raster image. For example, a first vector plane may correspond to a first capture object of the raster image, and a first vector line specified along the edge of the first vector plane may correspond to the edge of the first capture object. For example, a second vector plane may correspond to a second capture object of the raster image, and a second vector line specified along the edge of the second vector plane may correspond to the edge of the second capture object.

[0048] When 3D depth information is available, an electronic device can detect vector objects (3D vector objects) based on that information. Specifically, two or more planes can be included in a captured object representing a real 3D object. In this case, the two or more planes of a 3D captured object can be classified based on the 3D depth information. Therefore, the electronic device can vectorize the 3D captured object to generate two or more vector planes and vector lines associated with those planes. Thus, in another operation, different content can be mapped to each vector plane.

[0049] In operation 230, the electronic device can generate an editing screen by overlaying a vector image with a raster image.

[0050] The editing screen includes raster images and vector images. Within the editing screen, raster images and vector images can be overlaid on each other. Raster images and vector images can be overlaid to generate an edited image, and the edited image can be included as at least a portion of the editing screen.

[0051] Raster and vector images are separated into different manipulation objects and manipulated individually, but can be displayed as if overlaid on the editing screen. One or more vector lines and one or more vector planes included in the vector image can correspond to capture objects included in the raster image. Therefore, the editing screen is configured to overlay raster and vector images to visually represent the correspondence.

[0052] In operation 240, the electronic device can display a user interface representation, including an editing screen, on the display.

[0053] Electronic devices can overlay raster images and vector images and display them within a portion of a user interface representation. The user interface representation includes a drawing area and a content selection area. An editing screen is displayed in the drawing area. One or more selectable items are displayed in the content selection area. See below for reference. Figure 6 A more detailed example configuration of the drawing area and content selection area is provided.

[0054] Selectable content displayed in the content selection area can be represented as a representative image, such as a thumbnail or preview. Selectable content refers to content that an electronic device can map to at least one of vector planes or vector lines included in the editing screen. For example, vector objects such as vector planes or vector lines can be processed as content mapping areas, and selectable content can be associated with content mapping areas. The content associated with a content mapping area is the selected content and can form a content layer.

[0055] In operation 250, the electronic device can generate a projection screen in response to user input (e.g., focus input or selection input) to a user interface representation.

[0056] In the editing screen, vector lines or vector planes can be treated as content-mapped areas. One or more pieces of content can be mapped to a content-mapped area. Content that can be mapped to a content-mapped area can then be arranged in a content selection area.

[0057] In one embodiment, the electronic device can switch the content mapping area that received the focus input to a focused state based on receiving a focus input for any of the content mapping areas. In response to receiving a selection input for content while the content mapping area remains in focus, the electronic device can map the content receiving the selection input to the content mapping area that is in focus.

[0058] In one embodiment, the electronic device can switch the content to a focused state based on receiving a focus input for the content. The electronic device can also map the content in focus to the content mapping area where the selection input was received, in response to receiving a selection input for a content mapping area while the content remains in focus.

[0059] One or more pieces of content can be mapped to a content mapping area. One, two, or more pieces of content can be mapped to a single content mapping area. For example, a red flat pattern and a picture of a puppy can be mapped together to a vector plane, and in this case, the projection screen can be configured such that the red flat pattern and the picture of the puppy overlap each other on the vector plane. Similarly, a video of a puppy and a grassy background can be mapped together to a vector plane, and in this case, the projection screen can be configured such that the video of the puppy and the grassy background overlap each other on the vector plane.

[0060] Based on content being mapped to at least one content mapping region, an electronic device can generate a projection screen where content is mapped to the content mapping region. During the generation of the projection screen, a content layer can be generated in response to content being mapped to the content mapping region, and the content layer may include the content mapped to the content mapping region.

[0061] A content layer can be generated whenever content is mapped to a content mapping region. For example, content layer #1 can be generated in response to content #1 being mapped to content mapping region #1, and content layer #2 can be generated in response to content #2 being mapped to content mapping region #2.

[0062] Even when content is mapped to the same vector line or vector plane multiple times, a content layer can still be generated each time content is mapped to a content mapping area. For example, content layer #1 can be generated in response to content #1 being mapped to a predetermined vector plane, and content layer #2 can be generated in response to content #2 being mapped to the same vector plane.

[0063] A projection screen can be generated when multiple content layers overlap. A projection screen can include multiple content layers. Each of the multiple content layers can be generated in response to content being mapped to a content mapping area (e.g., corresponding to a vector line or vector plane). Electronic devices can generate projection screens by overlaying multiple content layers.

[0064] In operation 260, the electronic device can directly output the generated projection screen, or it can send the projection screen to another device to output the projection screen.

[0065] In an embodiment, an electronic device (e.g., a projector PJT) can output a generated projection screen to the real environment in which the raster image is acquired. The raster image of the captured object is a captured image of the real object in the real environment. Therefore, the content of at least one vector object mapped to the vector object corresponding to the captured object can be represented on the real object associated with the captured object. In this way, the electronic device can realize extended reality in the real environment by using the real object as a reflector to represent the content.

[0066] In an embodiment, an electronic device (e.g., a user equipment (UE)) can send the generated projection screen to a projector PJT, causing the projector PJT to output the generated projection screen. The projector PJT, upon receiving the projection screen, can output the projection screen to the real environment in which the raster image was acquired. The content of at least one vector object mapped to the vector object corresponding to the captured object can be represented on a real object associated with the captured object. Thus, the electronic device can realize extended reality in a real environment by using real objects as reflectors to represent content.

[0067] Thus, according to various embodiments of this disclosure, an electronic device can acquire a captured image including an object (captured object) via a camera and automatically divide the captured object into content-mappable vector planes or vector lines. Furthermore, whenever content is mapped to an automatically divided vector plane or vector line, the electronic device can edit the content based on each layer by generating new content layers. Additionally, the electronic device can generate a projection screen by overlaying and displaying content layers and projecting the generated projection screen in real time to identify in real-time how the projection result (projector output screen) behaves in the real environment.

[0068] Figure 3 This is a flowchart illustrating example image mapping methods according to various embodiments.

[0069] Reference Figure 3 The electronic device can be a user equipment (UE) or a projector unit (PJT). Each of the following operations can be performed by one or more processing circuitry units, which can be included in the user equipment (UE) or the projector unit (PJT).

[0070] Reference Figure 3 The image mapping method according to various embodiments of this disclosure may include operations 310 and 320. See reference... Figure 2 Operation 320 is executed before operation 210, and operations after operation 210 do not need to be repeated here.

[0071] Reference Figure 3 In operation 310, the electronic device can detect events.

[0072] Events can be preset by the user. Events may include, for example, events where an electronic device connects to another electronic device or events where a start command is received through an application. For example, events may include events where an electronic device (e.g., a user equipment UE (or a projector PJT)) is wired or wirelessly connected to another electronic device (e.g., a projector PJT (or a user equipment UE)), or events where an image mapping application executes commands including those for performing image mapping operations according to various embodiments of this disclosure.

[0073] In operation 320, the electronic device may output (e.g., display) a standby screen in response to the detection of an event. For example, the electronic device may send a command to allow another electronic device to output a standby screen, or it may send the standby screen to another electronic device. For example, the electronic device may directly project the standby screen.

[0074] Standby screens can include white screens and striped pattern screens. A white screen is a completely white screen. A striped pattern screen is a screen that includes a striped pattern. When a striped pattern screen is reflected onto a real object, the size or spacing between the stripes varies with depth, and the depth of the real object can be determined based on the differences in the size or spacing between the stripes.

[0075] The electronic device can selectively output a white screen or a striped pattern screen based on user input. For example, the electronic device can output a white screen to detect 2D vector objects and a striped pattern screen to detect 3D vector objects. The electronic device can output a white screen in response to receiving a first user input and output a striped pattern screen in response to receiving a second user input. The first user input and / or the second user input can be received directly through an input unit provided in the electronic device, or received from another electronic device in the form of an electrical signal or message.

[0076] Subsequently, as in operation 210, the electronic device can acquire a raster image of the actual object projected onto the standby screen. For example, a captured image of the actual object projected onto the standby screen can be acquired via a camera. The raster image can be substantially the same as the captured image, or it can be configured to be at least a portion of the captured image selected by user input.

[0077] In this way, the electronic device can detect the attributes (e.g., position or shape) of the real object to be mapped to a vector object before the output of the projection screen containing the content, by adding a projection standby screen before acquiring the raster image. Furthermore, the electronic device can provide the user with a projector output that adapts to the real environment by mapping the content to a vector object suitable for the real object, generated based on the detection results.

[0078] Figure 4 This is a signal flow diagram illustrating example image mapping methods according to various embodiments.

[0079] Reference Figure 4 User equipment (UE) can be with Figure 1 In response to the user equipment (UE), the projector PJT can be connected to... Figure 1 The corresponding projector PJT. The user equipment (UE) or projector PJT can be as described above. Figure 2 and Figure 3 The described electronic device.

[0080] In operation 410, the user equipment (UE) can be communicatively connected to the projector PJT. For example, the UE and the projector PJT can be connected to each other wired or wirelessly, and while maintaining the connection, the UE and the projector PJT can send electrical signals, messages, commands and / or data to or receive electrical signals, messages, commands and / or data from each other.

[0081] In operation 420, the projector PJT can project (e.g., display) a standby screen via an image projection unit. The standby screen may include a white screen or a dedicated standby display. In embodiments, the projector PJT can project the standby screen based on a wired or wireless connection to a user equipment UE. In embodiments, the projector PJT can project the standby screen in response to receiving a command from the user equipment UE instructing it to project the standby screen. In embodiments, the projector PJT can project the standby screen in response to receiving user input to its input unit.

[0082] In operation 430, the user equipment (UE) can acquire a raster image by capturing the real environment of the projected standby screen via a camera. The UE can acquire a captured image of the real environment. The captured image may include the real environment of the projected standby screen and the surrounding area of ​​the real environment of the projected standby screen. Here, the captured image is generated in the form of a raster image rather than a vector image.

[0083] In operation 440, the user equipment (UE) can generate a vector image based on the captured image. The vector image can be generated by a vectorization engine. The UE can use the vectorization engine to perform vectorization on at least a portion of the raster image. Vectorization can be performed on all or part of the raster image. When vectorization is performed on a portion of the raster image, the vector image can be formed with dimensions corresponding to that portion. The portion can be determined by user input. For example, the user can select a portion of the raster image by input, and vectorization can be performed on the selected portion.

[0084] Vector images can include vector objects. Vector objects can include vector lines and / or vector planes. Content can be mapped to vector objects.

[0085] In operation 450, the user equipment (UE) can generate an editing screen. The editing screen can include raster images and vector images. The vector images and raster images can overlap each other. Vector objects included in the vector image can correspond to captured objects included in the raster image. For example, vector objects can correspond one-to-one with captured objects at overlapping locations.

[0086] In the editing screen, vector objects can be treated as content-mapped areas. One or more pieces of content can be mapped to these content-mapped areas. The editing screen can display a raster image in a fixed position and map content to the content-mapped areas included in the vector image. Therefore, through the editing screen, users can pre-determine the form of content to be displayed for a real-world environment.

[0087] In operation 460, the user equipment (UE) can send the edit screen to the projector PJT. The UE can send the edit screen to the projector PJT at least periodically or in real time.

[0088] In Operation 465, the projector PJT can project an editing screen received from the user equipment (UE). Therefore, the user can recognize the editing screen through the real environment projected by the projector PJT, rather than through the UE's display. The user can visually identify the projection result where vector objects overlap with real objects; therefore, when there is a positional error between the vector and real objects, the user can directly adjust the position of the vector objects.

[0089] In various embodiments of this disclosure, operations 460 and 465 may be omitted.

[0090] In operation 470, the user equipment (UE) may display a user interface representation on a display. The user interface representation may include a drawing area, a content selection area, or multiple buttons. In the drawing area, raster images and vector images may be displayed overlapping each other. The drawing area may include a content mapping area corresponding to a vector object.

[0091] In operation 480, the user equipment (UE) can generate a projected screen based on user input (e.g., focus input, selection input, etc.) to a user interface representation. The UE can map content about the option to which the focus input has been applied to a content mapping area based on receiving focus input for an option arranged in a content selection area and receiving selection input for a content mapping area.

[0092] User equipment (UE) can generate content layers in response to mapping content to content mapping areas. Each time content is mapped to a content mapping area, the UE can generate a content layer. A piece of content can be included in a content layer. A projection screen can be generated by overlaying multiple content layers. A content layer can include one piece of content, and when multiple content layers overlap each other, the projection screen can include multiple pieces of content. Multiple pieces of content included in the projection screen can be displayed, executed, or reproduced independently.

[0093] User equipment (UE) may include interaction information between content layers in a projection screen. This interaction information includes information for interaction between content included in the content layers. For example, an image included in a first content layer and an image included in a second content layer may be synchronized with each other and reproduced. For example, an image in the second content layer may be reproduced in response to the completion of reproduction of an image in the first content layer.

[0094] In Operation 485, the User Equipment (UE) can send the generated projection screen to the Projector Projector (PJT). The UE can send the projection screen and can also send commands or messages to control the Projector Projector (PJT) to output the received projection screen.

[0095] In operation 490, the projector PJT can display the received projection screen. Specifically, if a projection screen is received, it can be output through the image projection unit. In response to receiving a projection screen or a command or message used to control the output of the projection screen, the projection screen can be output through the image projection unit.

[0096] Figure 5 This is a signal flow diagram illustrating example image mapping methods according to various embodiments.

[0097] Reference Figure 5 User equipment (UE) can be with Figure 1 In response to the user equipment (UE), the projector PJT can be connected to... Figure 1 The corresponding projector PJT. The user equipment (UE) or projector PJT can be as described above. Figure 2 and Figure 3 The described electronic device.

[0098] In operation 510, the user equipment (UE) can communicatively connect to the projector PJT. For example, the UE and the projector PJT can be connected to each other wired or wirelessly, and while maintaining the connection, the UE and the projector PJT can send electrical signals, messages, commands and / or data to or receive electrical signals, messages, commands and / or data from each other.

[0099] In operation 515, the projector PJT can project (e.g., display) a standby screen via an image projection unit. The standby screen may include a white screen or a dedicated standby display. In embodiments, the projector PJT can project the standby screen based on a wired or wireless connection to a user equipment UE. In embodiments, the projector PJT can project the standby screen in response to receiving a command from the user equipment UE instructing it to project the standby screen. In embodiments, the projector PJT can project the standby screen in response to receiving user input to the projector PJT's input unit.

[0100] In operation 520, the projector PJT can acquire a raster image by capturing the real environment at the projection standby screen via a camera. The projector PJT can acquire a captured image of the real environment. The captured image can include the real environment at the projection standby screen and the surrounding area of ​​the real environment at the projection standby screen. Here, the captured image is generated in the form of a raster image rather than a vector image.

[0101] In operation 525, the projector PJT can generate a vector image based on the captured image. The vector image can be generated by a vectorization engine. The projector PJT can use the vectorization engine to perform vectorization on at least a portion of the raster image. Vectorization can be performed on all or part of the raster image. When vectorization is performed on a portion of the raster image, the vector image can be formed with dimensions corresponding to that portion. The portion can be determined by user input. Specifically, the user can select a portion of the raster image by input, and vectorization can be performed on the selected portion.

[0102] Vector images can include vector objects. Vector objects can include vector lines and / or vector planes. Content can be mapped to vector objects.

[0103] In operation 530, the projector PJT can transmit at least one of a raster image and / or a vector image to the user equipment UE.

[0104] In operation 535, the user equipment (UE) can generate an edit screen. The edit screen may include raster images and vector images. The vector images and raster images may overlap each other. In one embodiment, the UE may use vector images and raster images received from a projector (PJT) to generate the edit screen. In another embodiment, the UE may use vector images received from a projector (PJT) and raster images acquired by a camera located in the UE to generate the edit screen.

[0105] Vector objects included in a vector image can correspond to captured objects included in a raster image. For example, vector objects can correspond one-to-one with captured objects at locations where they overlap.

[0106] In the editing screen, vector objects can be treated as content-mapped areas. One or more pieces of content can be mapped to these content-mapped areas. The editing screen can display a raster image in a fixed position and map content to the content-mapped areas included in the vector image. Therefore, through the editing screen, users can pre-determine the form of content to be displayed for a real-world environment.

[0107] In operation 540, the user equipment (UE) can send the edit screen to the projector PJT. The UE can send the edit screen to the projector PJT at least periodically or in real time.

[0108] In operation 545, the projector PJT can project (e.g., display) an editing screen received from the user equipment (UE). Therefore, the user can recognize the editing screen through the real environment projected by the projector PJT, rather than through the UE's display. The user can visually identify the projection result where the vector object overlaps with the real object; therefore, when there is a positional error between the vector object and the real object, the user can directly adjust the position of the vector object.

[0109] In various embodiments of this disclosure, operations 540 and 545 may be omitted.

[0110] In operation 550, the user equipment (UE) can display a user interface representation on a display. The user interface representation may include a drawing area, a content selection area, or multiple buttons. In the drawing area, raster images and vector images may be displayed overlapping each other. The drawing area may include a content mapping area corresponding to a vector object.

[0111] In operation 555, the user equipment (UE) can generate a projected screen based on user input (e.g., focus input, selection input, etc.) to a user interface representation. The UE can map content about the option to which the focus input has been applied to a content mapping area based on receiving focus input for an option arranged in a content selection area and receiving selection input for a content mapping area.

[0112] User equipment (UE) can generate a content layer in response to mapping content to a content mapping area. Each time content is mapped to a content mapping area, the UE can generate a content layer. A content layer can include a single piece of content. A projection screen can be generated by overlaying multiple content layers. A content layer can include a single piece of content, and when multiple content layers overlap each other, the projection screen can include multiple pieces of content. Multiple pieces of content included in the projection screen can be displayed, executed, or reproduced independently.

[0113] User equipment (UE) can include interaction information between content layers in the projection screen. This interaction information includes information for interaction between content included in the content layers. For example, an image included in a first content layer and an image included in a second content layer can be synchronized with each other and reproduced. For example, an image in the second content layer can be reproduced in response to the completion of reproduction of an image in the first content layer.

[0114] In operation 560, the User Equipment (UE) can send the generated projection screen to the Projector Phone (PJT). The UE can send the projection screen and can also send commands or messages to control the PJT to output the received projection screen.

[0115] In operation 565, the projector PJT can display the received projection screen. Specifically, if a projection screen is received, it can be output through the image projection unit. In response to receiving a projection screen or a command or message used to control the output of the projection screen, the projection screen can be output through the image projection unit.

[0116] Figure 6 This is a diagram illustrating example user interface representations according to various embodiments. Figure 7 This is a diagram illustrating a sub-user interface representation associated with a capture button according to various embodiments. Figure 8 This is a diagram illustrating a user interface representation associated with a free-drawn button according to various embodiments. Figure 9 This is a diagram illustrating a user interface representation associated with a pen button according to various embodiments.

[0117] Reference Figure 6 The user interface (MUI) can include a drawing area (DA), a content selection area (CA), or multiple buttons.

[0118] In the rendering area (DA), raster images and vector images can be represented as overlapping each other. The aspect ratio of the rendering area (DA) can be approximately the same as or similar to the output ratio of the projector.

[0119] Multiple content thumbnails (CTs) can be represented within the content selection area (CA). Content thumbnails can include preview images of content (e.g., selectable content) that can be mapped to the content mapping area. Content thumbnails can also be "options" set to receive user input.

[0120] Multiple buttons include a capture button B1, a freehand drawing button B2, a pen button B3, and / or a finish button B4. These buttons are configured to receive user input. They are also configured to perform specific actions in response to user input.

[0121] Reference Figure 6 In response to receiving user input on the capture button B1, the electronic device can invoke a sub-user interface representation (SUI). Specifically, in response to receiving user input (e.g., touch input) on the capture button B1, the electronic device can switch the user interface representation (MUI) to the sub-user interface representation (SUI) and display the sub-user interface representation (SUI) on the display. In this disclosure, the sub-user interface representation (SUI) can be understood as a "capture screen" for acquiring captured images through one or more cameras. The sub-user interface representation (SUI) can be any of the sub-user interface representations that depend on the user interface representation (MUI). References below... Figure 7 A sample configuration representing the sub-user interface (SUI) is described in more detail.

[0122] Reference Figure 6 and Figure 7 The sub-user interface representation (SUI) may include a first button B11 and / or a second button B12. For example, based on user input received on the first button B11, the electronic device may switch the SUI to a user interface representation (MUI) and display the MUI on the display. Similarly, based on user input received on the second button B12, the electronic device may perform a camera shot or assign a captured image obtained by the camera to the drawing area DA of the MUI. In response to receiving user input on the second button B12 before acquiring the captured image, the electronic device may acquire the captured image by performing a camera shot. Furthermore, in response to receiving user input on the second button B12 after acquiring the captured image, the electronic device may input the captured image into the drawing area DA of the MUI.

[0123] Reference Figure 6 and Figure 8 In response to receiving user input on the free-draw button B2, the electronic device can switch to free-draw mode. Free-draw mode is any of the various execution modes of the electronic device.

[0124] While maintaining free-drawing mode, the electronic device can receive user input to the drawing area DA and can generate a content mapping area based on the user input. For example, a drawing line DL corresponding to the user input (e.g., drag input) to the drawing area DA can be generated, and the content mapping area can be specified through the drawing line DL. Content can be mapped to the content mapping area specified by the drawing line DL. A thumbnail of the content that can be mapped to the content mapping area can be displayed in the content selection area CA.

[0125] While maintaining free drawing mode, the free drawing button B2 can be represented by different colors. Free drawing mode can be enabled or disabled in response to selection input on the free drawing button B2. For example, based on receiving user input on the free drawing button B2 while maintaining free drawing mode, the electronic device can disable free drawing mode. Conversely, in response to receiving user input on the free drawing button B2 while free drawing mode is disabled, the electronic device can enable free drawing mode.

[0126] Reference Figure 6 and Figure 9 In response to receiving user input on the pen button B3, the electronic device can switch to pen mode. Pen mode is any of the various execution modes of the electronic device.

[0127] While maintaining pen mode, the electronic device can receive user input to the drawing area DA and can display a pen representation BR based on the user input. For example, a pen representation BR corresponding to user input (e.g., drag input) to the drawing area DA can be generated and displayed in the drawing area DA.

[0128] While remaining in brush mode, the electronic device can display a brush toolbox BT for determining the style of the brush representation BR. The brush toolbox BT can include various types of brush options. The electronic device can set the style of the brush representation BR based on the selection input of the brush options. For example, the electronic device can set the style of the brush representation BR, including at least one of the brush thickness, color, and / or pattern of the brush representation BR.

[0129] Return to reference Figure 6 For example, in response to receiving user input to the complete button B4, the electronic device can terminate the editing screen and control the output of the editing results to the image projection unit of the electronic device or an image projection unit located in another electronic device.

[0130] Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 14 , Figure 15 and Figure 16 This is a diagram illustrating changes in the user interface representation according to various embodiments of the image mapping method.

[0131] Figure 10 , Figure 11 , Figure 12 , Figure 13 , Figure 14 , Figure 15 and Figure 16 The mobile device screen represents the user interface (MUI), and the projector output screen (PDS) represents the result of projecting the screen onto the real environment. The mobile device screen is similar to the one referenced above. Figure 6 The mobile device screens described are substantially the same. Since the projector output screen PDS is obtained by projecting the projector screen onto the real environment, it may include real objects RO located in the real environment (e.g., including a first real object RO1 and a second real object RO2). In this disclosure, the projector output screen PDS can be used interchangeably with "projector output result", "projector projection result" and "projected real environment".

[0132] Reference Figure 10 The projector will not project the editing screen or the projection screen until the editing screen is generated.

[0133] However, this is not the only limitation. The projector can output a standby screen SBS for generating the editing screen until the editing screen is generated. The standby screen SBS can be a white screen or a striped pattern screen. The output standby screen SBS can be reflected by a real object RO, and the light reflected from the real object RO can be captured by the user device's camera.

[0134] The projector can output an editing guide NT before generating the editing screen. The editing guide NT can include, for example, text messages such as "Editing LightWARP in real time".

[0135] In response to receiving user input to a capture button for the user interface representation MUI, the user device switches the user interface representation MUI to the sub-user interface representation SUI.

[0136] Reference Figure 11 While the user device displays the sub-user interface (SUI), the projector does not project a projection screen. However, it is not limited to this; the projector can output a standby screen (SBS) for generating an editing screen.

[0137] In this embodiment, in response to receiving user input to the capture button B1, the user equipment sends a message to the projector to control the output of the standby screen SBS. The user equipment can receive light reflected from the real object RO at the projected standby screen SBS via a camera and can generate a captured image CI from the light.

[0138] The standby screen (SBS) can be a white screen or a striped pattern screen.

[0139] In this embodiment, while displaying the sub-user interface (SUI), the user equipment can perform a camera capture operation in response to receiving user input to the second button. Therefore, the camera can acquire a captured image CI of the real environment corresponding to the projector output screen (PDS). The captured image CI may include a real object RO. Specifically, the captured image CI may include a real object RO at the projection standby screen (SBS).

[0140] Reference Figure 12Based on the captured image CI, the captured image CI can be displayed as at least a part of the sub-user interface representing the SUI. The captured image CI includes a projected image PI corresponding to the projector output screen PDS. The projected image PI can include a captured object CO. The captured object CO corresponds to the real object RO. The captured object CO may have a different color or pattern than the real object RO. The captured object CO is the result of the standby screen SBS being projected and reflected on the real object RO, and the real object RO may differ in color or pattern representation. For example, the captured object CO may appear brighter than the real object RO. For example, unlike the real object RO, the captured object CO may have a striped pattern.

[0141] When the captured image CI is acquired, the user equipment can provide the captured image CI as a user interface representation (MUI) in response to receiving user input to the second button B12. Specifically, the user equipment can input the captured image CI into the drawing area DA of the user interface representation MUI.

[0142] An edit point P can be displayed at the vertex of the captured image CI. Based on user input regarding the edit point P, the user device can determine the editing area. The editing area can be a polygon defined by the edit point P. The polygon can include, but is not limited to, triangles, squares, pentagons, or hexagons.

[0143] Based on the user input received to the second button B12, the user equipment can associate the captured image CI included in the editing area with the drawing area DA of the user interface representation MUI.

[0144] Reference Figure 13 The user equipment can adjust the size and / or position of the editing area based on user input to the edit point P. For example, the position of the edit point P can be changed by dragging it. As the position of the edit point P is changed, at least one of the size and / or position of the editing area can be adjusted.

[0145] Based on the received user input to the second button B12, the user equipment can associate the image corresponding to the adjusted editing area with the drawing area DA of the user interface representation MUI. The image corresponding to the editing area can then be displayed in the drawing area DA.

[0146] Reference Figure 14 The user device can display an image in the drawing area (DA) corresponding to the editing area representing the sub-user interface (SUI). The image displayed in the drawing area (DA) can be a combination of raster and vector images superimposed on each other.

[0147] For example, based on user input received to the second button B12 in the sub-user interface representation (SUI), the user equipment can apply vectorization to the raster image corresponding to the editing area. The user equipment can acquire a vector image of the raster image corresponding to the editing area based on the vectorization. The user equipment can overlay the raster image and the vector image and display them in the drawing area (DA).

[0148] Vectorization is not limited to being performed in response to user input to the second button B12 representing the SUI sub-user interface. For example, vectorization can be performed based on the acquired image CI. When vectorization is performed based on the acquired image CI, Figure 12 and Figure 13 The captured image CI shown can be displayed as an overlay with a vector image vectorized from the captured image CI.

[0149] A vector image may include vector objects VO (e.g., a first vector object VO1 and a second vector object VO2). Vector objects VO may correspond to captured objects CO and / or real objects RO. Vector objects VO may include vector lines VL and / or vector planes VP. Each of the vector lines VL and / or vector planes VP can be processed as a content-mapped region.

[0150] Reference Figure 15 In response to receiving user input (e.g., focus input) on a content mapping area, the selected content mapping area is switched to a focus state. While maintaining the focus state, in response to receiving selection input for a content option CT, the user equipment can assign content associated with the content option CT at the point of selection to the content mapping area in the focus state. Therefore, content can be mapped to the focused content mapping area.

[0151] While remaining in focus, the content-mapped area can be represented as a focus pattern. In addition to the focus state, the focus pattern can be represented with a different color and / or pattern than the content-mapped area.

[0152] Reference Figure 16 The user device maps content to a content mapping region. A content layer is generated in response to content being mapped to a content mapping region. The content layer includes a single piece of content. In response to mapping content to a content mapping region, a content layer is generated that includes the content mapped to the content mapping region.

[0153] User devices generate a projection screen by overlaying content layers. For example, a user device can generate a projection screen whenever a content layer is generated. Alternatively, a user device can generate a projection screen by overlaying previously generated content layers with a newly generated content layer.

[0154] The user device sends a projection screen to the projector. The user device sends a generated projection screen to the projector in response to the generation of the projection screen. For example, whenever a content layer is generated, the user device can generate a projection screen by overlaying a previously generated content layer with a newly generated content layer, and can send the generated projection screen to the projector. In response to, for example, a new content layer being added to a new projection screen compared to a previous projection screen, or a previous content layer being edited (or deleted), the user device can send a new projection screen to the projector.

[0155] The projector can project a projection screen through the image projection unit. For example, whenever a new projection screen is received from the user device, the projector can project the new projection screen through the image projection unit. Therefore, whenever a new content layer is created, the projector can project a new projection screen through the image projection unit.

[0156] In this embodiment, the user equipment can share the image displayed through the drawing area DA with the projector. For example, the user equipment can send the image displayed through the drawing area DA to the projector, either at least in real time or continuously. Therefore, when editing the image displayed in the drawing area DA of the user equipment, the projector can output a projection screen reflecting the editing.

[0157] The content layer that forms the projection screen of a projector includes content in the content mapping area corresponding to vector objects VO. The vector objects VO correspond to real objects RO in position and / or shape. Therefore, if the projection screen is output by a projector, the content included in the content layer can be represented on real objects RO in the real environment.

[0158] For example, if content is mapped to a first vector object VO1, a content layer is generated for the first vector object VO1. The user device provides a projection screen including the generated content layer to a projector, and the projector outputs the projection screen. Since the first vector object VO1 has a position and / or shape corresponding to a first real object RO1 in the real environment, content can be represented on the first real object RO1.

[0159] Figure 17 This is an exploded view showing an example configuration of a projection screen according to various embodiments. Figure 18 This illustrates various embodiments. Figure 17 A diagram showing the projection result on the projection screen.

[0160] Reference Figure 17 A projection screen (PS) can include multiple content layers. The projection screen (PS) can be configured by stacking, for example, a first content layer (L1), a second content layer (L2), and a third content layer (L3).

[0161] The first content layer L1, the second content layer L2, and the third content layer L3 can contain substantially the same or different content. The content included in the first content layer L1, the second content layer L2, and the third content layer L3 can be configured by mapping the content to any of the first vector object VO1, the second vector object VO2, and the third vector object VO3.

[0162] For example, a first content layer L1 can be configured by mapping a first content C1 to a first vector object VO1. A second content layer L2 can be configured by mapping a second content C2 to a second vector object VO2. A third content layer L3 can be configured by mapping a third content C3 to a third vector object VO3.

[0163] The projection screen PS can be configured by overlaying a first content layer L1, a second content layer L2, and a third content layer L3. For example, the projection screen PS can be generated to represent the first content C1 in the content mapping area corresponding to the first vector object VO1, the second content C2 in the content mapping area corresponding to the second vector object VO2, and the third content C3 in the content mapping area corresponding to the third vector object VO3.

[0164] Reference Figure 18 The projector PJT projects the projection screen PS onto the projection area R1 via the image projection unit OP. The projection screen PS is generated to correspond to the projection area R1, and the projection screen PS is not projected onto the outer area R2. The real object RO located in the projection area R1 and the vector object (e.g., including...) that forms the basis of the projection screen PS... Figure 17 The first vector object VO1, the second vector object VO2, and the third vector object VO3 correspond to each other, and the content mapped to the corresponding vector object is represented in each real object RO.

[0165] The projection area R1 can be identified by the user equipment (UE) using the standby screen. The projection screen PS and the standby screen (e.g., Figure 11 and Figure 12 The standby screen (SBS) can be projected to have a substantially identical or similar area and position. While the projector PJT is projecting the standby screen, the user equipment (UE) can also capture an image including the standby screen by taking a picture of the projection area R1 of the projected standby screen.

[0166] User equipment (UE) can use feature information (e.g., including at least one of brightness, markings, and / or stripe patterns) to identify the standby screen included in the captured image. For example, the UE can identify markings included as at least a portion of the standby screen and identify the projection area R1 based on at least one of the identified markings' position and / or number. Similarly, the UE can identify stripe patterns included as at least a portion of the standby screen and identify the projection area R1 based on the stripe pattern identification. For example, the UE can identify the projection area R1 based on the brightness difference in a portion of the captured image caused by the standby screen projecting white.

[0167] The identification of the projection area R1 described above can be implemented not only in the user equipment (UE) but also in the projector PJT. For example, the projector PJT may include one or more cameras for generating captured images, and the captured images can be acquired using the cameras. In this case, the projector PJT itself can also use the obtained captured images to identify the projection area R1.

[0168] The projection screen PS can be provided by the user equipment (UE) or generated by the projector PJT.

[0169] Figure 19 This is a diagram illustrating frame composition of a projection screen according to various embodiments.

[0170] Reference Figure 19 A projection screen PS can be generated by overlaying multiple content layers (e.g., including a first content layer L1, a second content layer L2, a third content layer L3, a fourth content layer L4, and a fifth content layer L5).

[0171] The content mapped to each of the first content layer L1, the second content layer L2, the third content layer L3, the fourth content layer L4, and / or the fifth content layer L5 can be any of a variety of content types. Content types may include, for example, video, images, live content, and patterns, but are not limited to these.

[0172] In this embodiment, the video may include repeatedly played image frames. The video may also include image frames that are played only within a specific time period. An image may include a single frame. An image comprising a single frame is a still image and cannot express animation effects. Real-time content contains continuously generated image frames. A pattern may include a single frame. A pattern may be included as part of an image.

[0173] Real-time content is content that allows users to receive feedback in real time and includes newly generated image frames as users interact with the content. Real-time content can include, for example, live news on chat services, sports broadcasts, and streaming platforms.

[0174] A projection screen PS can be generated by overlaying (or compositing) multiple content layers. A projection screen PS can be configured by compositing frames containing all content layers (e.g., first content layer L1, second content layer L2, third content layer L3, fourth content layer L4, and fifth content layer L5).

[0175] In this way, the entire frame of the projection screen PS can be set by compositing the individual frames of the content layer of the projection screen. Therefore, while reproducing the entire frame of the projection screen, each content layer can be reproduced separately based on each frame.

[0176] Figure 20 This is a block diagram illustrating example configurations of a user equipment according to various embodiments.

[0177] Reference Figure 20 According to the example user equipment (UE) Figure 1 The user equipment (UE) may include at least one processor (e.g., including processing circuitry) 2010, an input unit (e.g., including input circuitry) 2020, an output unit (e.g., including output circuitry) 2030, a communication unit (or transceiver) (e.g., including communication circuitry) 2040, a memory 2050, and / or a camera 2060. The output unit 2030 may include a display unit (e.g., including a display), such as a display for providing visual information to the user.

[0178] Processor 2010 may include various processing circuitry and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of the at least one processor may be individually and / or collectively configured in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, cases where one processor performs some of the described functions while another processor performs other functions, and cases where a single processor can perform all the described functions. Additionally, at least one processor may include, for example, a combination of processors performing various described / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions. Processor 2010 may execute, for example, software (e.g., a program) to control at least one other component (e.g., a hardware component or a software component) of user equipment 120 connected to processor 2010, and may process or compute various data. As at least part of data processing or computation, processor 2010 may store commands or data received from another component (e.g., input unit 2020) in memory 2050, process the commands or data stored in memory 2050, and store the result data in memory 2050. Processor 2010 may include a main processor (e.g., a central processing unit (CPU) or application processor (AP)) or an auxiliary processor (e.g., a graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) that may operate independently of or in conjunction with the main processor. For example, when a user equipment (UE) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use lower power than the main processor or designated for a specific function. The auxiliary processor may be implemented separately from the main processor or as part of the main processor.

[0179] Memory 2050 may store various data used by at least one component of the user equipment (UE), such as processor 2010. The various data may include, for example, software (e.g., a program) and input or output data for commands associated with it. Memory 2050 may include volatile memory or non-volatile memory.

[0180] Input unit 2020 may include various circuits and receive commands or data from outside the user equipment (UE) (e.g., from a user) to be used by components of the user equipment (UE) (e.g., processor 2010). Input unit 2020 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0181] Output unit 2030 may include various circuits and output visual or auditory information to an external source that will be transmitted to a user. Output unit 2030 may include, for example, a speaker or receiver capable of outputting auditory information. The speaker may be used for general purposes, such as playing multimedia or playing records. The receiver may be used to receive incoming calls. The receiver may be implemented separately from the speaker or as part of the speaker. Output unit 2030 may include a display unit. The display unit may include a display that outputs visual information to an external source. The display unit may include, for example, a monitor, a holographic device, or a projector, and control circuitry for controlling a corresponding one of the monitor, holographic device, and projector. The display unit may include a touch sensor configured to detect touch, or a pressure sensor configured to measure the intensity of the force generated by the touch.

[0182] The communication unit 2040 may include various communication circuits and may be referred to as a transceiver. The communication unit 2040 may support communication between the user equipment (UE) and external electronic devices (e.g., ...). Figure 1 The communication unit 2040 establishes a direct (e.g., wired) or wireless communication channel with the service server 110 and performs communication via the established communication channel. The communication unit 2040 may include one or more communication processors that can operate independently of the processor 2010 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. The communication unit 2040 may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a Global Navigation Satellite System (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding one of these communication modules can communicate via a first network (e.g., a short-range communication network, such as Bluetooth). TM Communication modules can communicate with external electronic devices via Wi-Fi Direct, Infrared Data Association (IrDA), or a second network (e.g., long-range communication networks such as traditional cellular networks, 5G networks, next-generation communication networks, the Internet, or computer networks (e.g., local area networks (LANs) or wide area networks (WANs)). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components that are separate from each other (e.g., multiple chips).

[0183] Camera 2060 can acquire images of the user or the surrounding environment, and the acquired images can serve as the basis for interactions such as recognizing user movement or facial features, or recognizing changes in the surrounding environment. Camera 2060 can transmit the acquired images to another component, such as processor 2010 or memory 2050. Methods or operations according to various embodiments of this disclosure can be implemented based on the images acquired by camera 2060.

[0184] Figure 21 This is a block diagram illustrating example configurations of a projector according to various embodiments.

[0185] Reference Figure 21 According to the example projector PJT ( Figure 1 The projector (PJT) may include at least one processor (e.g., including processing circuitry) 2110, an input unit (e.g., including input circuitry) 2120, an output unit (e.g., including output circuitry) 2130, a communication unit (e.g., including communication circuitry) 2140, a memory 2150, a camera 2160, and / or an image projection unit (e.g., including an image projector) 2170. The output unit 2130 may include a display unit (e.g., including a display) corresponding to the component, such as a display for providing visual information to a user.

[0186] Processor 2110 may include various processing circuitry and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of the at least one processor may be configured individually and / or in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, cases where one processor performs some of the described functions while another processor performs other functions of the other described functions, and cases where a single processor can perform all the described functions. Additionally, at least one processor may include, for example, a combination of processors performing various described / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions. Processor 2110 may execute, for example, software (e.g., a program) to control at least one other component (e.g., a hardware component or a software component) of user equipment 121 connected to processor 2110, and may process or compute various data. As at least part of data processing or computation, processor 2110 may store commands or data received from another component (e.g., input unit 2120) in memory 2150, process the commands or data stored in memory 2150, and store the result data in memory 2150. Processor 2110 may include a main processor (e.g., a central processing unit (CPU) or application processor (AP)) or an auxiliary processor (e.g., a graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)) that may operate independently of or in conjunction with the main processor. For example, when a projector PJT includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use less power than the main processor or designated for a specific function. The auxiliary processor may be implemented separately from the main processor or as part of the main processor.

[0187] Memory 2150 may store various data used by at least one component of the projector PJT (e.g., processor 2110). The various data may include, for example, software (e.g., a program) and input or output data of commands associated therewith. Memory 2150 may include volatile memory or non-volatile memory.

[0188] Input unit 2120 may include various circuits and receive commands or data from outside the projector PJT (e.g., a user) for use by components of the projector PJT (e.g., processor 2110). Input unit 2120 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0189] Output unit 2130 may include various circuits and output visual or auditory information to an external source that will be transmitted to a user. Output unit 2130 may include, for example, a speaker or receiver capable of outputting auditory information. The speaker may be used for general purposes, such as playing multimedia or playing records. The receiver may be used to receive incoming calls. The receiver may be implemented separately from the speaker or as part of the speaker. Output unit 2130 may include a display unit. The display unit may include a display that outputs visual information to an external source. The display unit may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a corresponding one of the display, holographic device, and projector. The display unit may include a touch sensor configured to detect touch, or a pressure sensor configured to measure the intensity of the force generated by a touch.

[0190] Communication unit 2140 may include various communication circuits and may be referred to as a transceiver. Communication unit 2140 may support communication between the projector PJT and external electronic devices (e.g., Figure 1 The communication unit 2140 establishes a direct (e.g., wired) or wireless communication channel with the service server 110 and performs communication via the established communication channel. The communication unit 2140 may include one or more communication processors that can operate independently of the processor 2110 (e.g., an application processor (AP)) and support direct (e.g., wired) or wireless communication. The communication unit 2140 may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a Global Navigation Satellite System (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding one of these communication modules can communicate via a first network (e.g., a short-range communication network, such as Bluetooth). TM Communication modules can communicate with external electronic devices via Wi-Fi Direct, Infrared Data Association (IrDA) or a second network (e.g., long-range communication networks such as traditional cellular networks, 5G networks, next-generation communication networks, the Internet, or computer networks (e.g., local area networks (LANs) or wide area networks (WANs)). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components that are separate from each other (e.g., multiple chips).

[0191] Camera 2160 can acquire images of the user or the surrounding environment, and the acquired images can serve as the basis for interactions such as recognizing user movement or facial features, or recognizing changes in the surrounding environment. Camera 2160 can transmit the acquired images to another component, such as processor 2110 or memory 2150. Methods or operations according to various embodiments of this disclosure can be implemented based on the images acquired by camera 2160.

[0192] Image projection unit 2170 may include a projector, which includes, for example, a light source and a lens. A light source can refer to an element that outputs light. The light output from the light source can be projected onto a screen (or wall) through the lens. Image projection unit 2170 can project an image provided by processor 2110 onto a front screen or wall, and a user can view the projected image.

[0193] The electronic device according to various embodiments can be one of a variety of types of electronic devices. The electronic device may include, for example, a display device, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a home appliance, etc. The electronic device according to the embodiments is not limited to those described above.

[0194] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the particular embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. As used herein, the singular forms “a,” “an,” and “described” are intended to also include the plural forms unless the context clearly indicates otherwise. As used herein, the term “and / or” should be understood to include any and all possible combinations of one or more of the listed items. As used herein, the terms “comprising,” “having,” and “including” are used only to indicate the presence of the features, components, parts, or combinations thereof described herein, but the use of such terms does not preclude the possibility of the presence or addition of one or more other features, components, parts, or combinations thereof. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” can include all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as “first” and “second” or “first” and “second” can be used to simply distinguish one component from another without limiting the components in other respects (e.g., importance or order).

[0195] As used herein, the terms "component" or "module" can include units implemented in hardware, software, or firmware, or any combination thereof, and are used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A component or module can be a single integral part or its smallest unit or portion adapted to perform one or more functions. For example, according to an embodiment, a "component" or "module" can be implemented in the form of an application-specific integrated circuit (ASIC).

[0196] As used in various embodiments of this disclosure, depending on the context, the term "if" may be interpreted as "when," "in response to," "in response to determination," or "in response to detection." Similarly, depending on the context, "if A is determined" or "if A is detected" may be interpreted as "when A is determined" or "in response to determination," or "when A is detected" or "in response to detection."

[0197] The program executed by the electronic device described herein can be implemented as a hardware component, a software component, and / or a combination thereof. The program can be executed by any system capable of executing computer-readable instructions.

[0198] Software can include computer programs, code, instructions, or a combination of one or more of these, and can configure processing devices to operate as needed, or can instruct processing devices independently or jointly. Software can be implemented as a computer program including instructions stored in a computer-readable storage medium. Computer-readable storage media can include, for example, magnetic storage media (e.g., read-only memory (ROM), random access memory (RAM), floppy disk, hard disk, etc.) and optically readable media (e.g., CD-ROM or digital versatile disc (DVD)). Furthermore, computer-readable storage media can be distributed across computer systems connected via a network, and computer-readable code can be stored and executed in a distributed manner. Computer programs can be distributed (e.g., downloaded or uploaded) directly between two user devices (UEs) via an app store (e.g., PlayStore™), or published online (e.g., downloaded or uploaded). If published online, at least a portion of the computer program product can be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

[0199] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities. Some of the multiple entities may be located in different components. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component can still perform one or more functions of each of the multiple components in the same or similar manner as the corresponding components in the multiple components before integration. According to various embodiments, operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

[0200] While this disclosure has been shown and described with reference to various exemplary embodiments, it will be understood that these exemplary embodiments are intended to be illustrative and not restrictive. Those skilled in the art will further understand that various changes in form and detail may be made without departing from the full scope of this disclosure, including the appended claims and their equivalents. It will also be understood that any embodiments described herein may be used in conjunction with any other embodiments described herein.

Claims

1. An electronic device comprising: monitor; transceiver; One or more memory units; as well as At least one processor, including processing circuitry, is electrically connected to the display, the transceiver, and the one or more memories, wherein the at least one processor is configured, individually or collectively, to: Use one or more cameras to capture images of one or more real-world objects; The projection region, which is included as at least a portion of the captured image, is identified based on feature information included in the captured image; A first image corresponding to the projection area is obtained from the captured image; The second image is generated by vectorizing the first image; An editing screen is generated by overlaying a second image onto a first image; Control the display to show a user interface representation including the editing screen; In response to one or more inputs to the user interface representation, a projection screen is generated; and The transceiver is controlled to send the projection screen to an external electronic device connected to the electronic device.

2. The electronic device according to claim 1, wherein, The first image includes a raster image comprising one or more captured objects corresponding to the one or more real objects, and the second image includes a vector image comprising one or more vector objects corresponding to the one or more captured objects.

3. The electronic device according to claim 2, wherein, The one or more vector objects include one or more vector lines or one or more vector planes.

4. The electronic device according to claim 2, wherein, The at least one processor is configured, individually and / or collectively, to: The one or more vector objects are processed into one or more content mapping regions; as well as In response to one or more inputs, generate one or more content layers having content mapped to one or more content mapping regions, and The projection screen includes the generated one or more content layers.

5. The electronic device according to claim 4, wherein, The projection screen is generated by overlaying one or more content layers.

6. The electronic device according to claim 1, wherein, The user interface includes a drawing area and a content selection area, and The editing screen is displayed in the drawing area, and the content options that can be mapped to the content mapping area are displayed in the content selection area.

7. The electronic device according to claim 6, wherein, The at least one processor is configured, individually and / or collectively, to generate a content layer having the content mapped to the content mapping region in response to receiving an input for mapping content associated with at least one of the content options in the content mapping region.

8. The electronic device according to claim 7, wherein, The at least one processor is configured, individually and / or collectively, to generate the content layer whenever it receives input for mapping content associated with at least one of the content options in the content mapping area.

9. The electronic device according to claim 7, wherein, The at least one processor is configured, individually and / or collectively, to: A new projection screen is generated by adding the generated content layer to a pre-generated projection screen; and In response to the generation of the new projection screen, the transceiver is controlled to send the new projection screen to the external electronic device.

10. The electronic device according to claim 1, wherein, The at least one processor is configured, individually and / or collectively, to control the transceiver to transmit messages for controlling the external electronic device to output a standby screen. The first image includes a raster image, which comprises a captured image of one or more real objects projected onto the standby screen.

11. The electronic device of claim 10, wherein, The standby screen includes a striped pattern screen.

12. The electronic device according to claim 11, wherein, The at least one processor is configured, individually and / or collectively, to: Three-dimensional depth information is obtained based on the stripe pattern represented in the captured image; Detecting 3D vector objects based on the aforementioned 3D depth information; and The 3D vector object is processed into a content-mapped region.

13. The electronic device according to claim 1, wherein, The entire frame of the projection screen is configured by synthesizing individual frames of one or more content layers of the projection screen. The entire frame of the projection screen is reproduced based on the individual frames of the one or more content layers.

14. An electronic device comprising: monitor; One or more image projectors; One or more memory units; as well as At least one processor, including processing circuitry, is electrically connected to the display, the one or more image projectors, and the one or more memories, wherein the at least one processor is configured, individually or collectively, to: Use one or more cameras to capture images of one or more real-world objects; The projection region included as at least a portion of the captured image is identified based on the feature information included in the captured image; A first image corresponding to the projection area is obtained from the captured image; The second image is generated by vectorizing the first image; An editing screen is generated by overlaying a second image onto a first image; Control the display to show a user interface representation including the editing screen; In response to one or more inputs to the user interface representation, a projection screen is generated; and The projection screen is projected using one or more image projectors.

15. An image mapping method performed by an electronic device, comprising: Use one or more cameras to capture images of one or more real-world objects; The projection region included as at least a portion of the captured image is identified based on the feature information included in the captured image; A first image corresponding to the projection area is obtained from the captured image; The second image is generated by vectorizing the first image; An editing screen is generated by overlaying a second image onto a first image; Display a user interface representation, including the editing screen, on the monitor; A projection screen is generated in response to one or more inputs represented by the user interface; as well as The projection screen is sent to an external electronic device connected to the electronic device.

Images