Apparatus, medium, and method for augmented reality guided mural creation

Abstract

The method includes detecting, via a sensor, a physical surface within a line of sight of the sensor. The method also includes providing a virtual surface corresponding to the physical surface. The method also includes receiving an image from a source other than the sensor. The method also includes dividing the image into multiple portions. The method also includes displaying placement indicators within the virtual surface that indicate where to place a physical object corresponding to one of the multiple portions of the image.

Description

[Background technology]

[0001] A user may wish to create a mural as an aesthetically appealing work of art. The advent of sticky paper notes, such as Post-it® Notes, has allowed users to place multiple notes on a wall and attempt to create a mural. However, creating such a mural on a wall can be a difficult and time-consuming task. Summary of the Invention

[0002] In one embodiment, one or more non-transitory computer-readable media are encoded with instructions that, when executed, configure a computing device to detect, via a sensor, a physical surface within a line of sight of the sensor. The one or more non-transitory computer-readable media further include instructions for providing a virtual surface corresponding to the physical surface and receiving an image from a source other than the sensor. The one or more non-transitory computer-readable media further include additional instructions for dividing the image into multiple portions and displaying placement indicators within the virtual surface that respectively indicate where to place a physical object corresponding to one of the multiple portions of the image.

[0003] In another embodiment, a method includes detecting, via a sensor, a physical surface within a line of sight of the sensor. The method also includes providing a virtual surface corresponding to the physical surface. The method also includes receiving an image from a source other than the sensor. The method also includes dividing the image into multiple portions. The method also includes displaying placement indicators within the virtual surface, each indicating a location for placing a physical object corresponding to one of the multiple portions of the image.

[0004] In yet another embodiment, a computing device includes a sensor configured to detect a physical surface within a line of sight of the sensor, and a memory. The computing device also includes a processor coupled to the memory and the sensor, the processor configured to provide a virtual surface corresponding to the physical surface, receive an image from a source other than the sensor, and divide the image into multiple portions. The computing device further includes display hardware coupled to the processor and the memory, the display hardware configured to display placement indicators within the virtual surface that respectively indicate where on the physical surface to place a physical object corresponding to one of the multiple portions of the image.

[0005] These and additional features provided by the presently disclosed embodiments will be more fully understood in conjunction with the following detailed description and drawings. [Brief explanation of the drawings]

[0006] The embodiments shown in the drawings are illustrative and are not intended to limit the subject matter defined by the claims.

[0007] [Figure 1] FIG. 1 illustrates a schematic diagram of a design selection interface according to an embodiment described herein.

[0008] [Figure 2] FIG. 2 is a schematic illustration of a mural design interface according to embodiments described herein.

[0009] [Figure 3] FIG. 3 is a schematic diagram illustrating augmented reality mural creation according to embodiments described herein.

[0010] [Figure 4]FIG. 4 is a flowchart of augmented reality mural creation according to embodiments described herein.

[0011] [Figure 5] FIG. 5 is a block diagram of computing hardware of a mobile device that may be used to implement the embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] This disclosure describes techniques that utilize augmented reality to assist users in partially recreating digital works in physical form. In some embodiments, a user may wish to create a mural based on a selected design. Rather than simply individually placing different colored Post-it® notes on a wall (which can be time-consuming and potentially inaccurate), the user can, for example, utilize augmented reality (AR) to identify or designate a virtual surface superimposed on a physical surface (e.g., a wall). Using the virtual surface, the user can visualize the virtual surface divided into a grid corresponding to (e.g.,) Post-it® notes. The user is advantageously assisted in creating the mural by displaying indicators (e.g., numbers) that guide the user to use one color for one number and another color for another number. In this way, AR allows the user to create a mural more accurately and quickly without worrying about where each Post-it® note fits into the overall mural composition. Instead, the user can literally create by the numbers.

[0013] Referring to FIG. 1 , the design selection interface 100 is shown in an AR environment; however, any suitable environment can be used, such as, but not limited to, virtual reality (VR), mixed reality (MR), or augmented reality (XR). In this embodiment, AR overlays virtual data, metadata, and the like onto a real-world environment, while VR represents the entire environment (or nearly the entire environment) virtually. In this embodiment, MR can be considered to be capable of performing both AR and VR, while XR can be considered to provide all the functionality of AR, VR, and / or MR, blending the physical and virtual worlds beyond the visual context. In this embodiment, a wearable headset is shown, but any suitable technology can be used, such as electronic glasses, holograms / projections, two-dimensional and / or three-dimensional displays on devices such as smartphones, laptops, desktop computers, and tablets. The design selection interface 100 may be displayed as, but is not limited to, a floating interface within the AR environment.

[0014] The design selection interface 100 may utilize any type of graphical or other interface for selecting or specifying the design 102. The design 102 may be any type of image (e.g., jpg, gif, png, vector graphic, etc.) created electronically or captured from the physical world. In some embodiments, the design 102 may be a simplified image with reduced resolution. A user may upload or capture an image using an image capture device. This image capture device may include, but is not limited to, a sensor mounted on an AR headset. In this embodiment, the sensor may include a camera, a LIDAR sensor, a time-of-flight (TOF) sensor, an ultrasonic sensor, a sonar, an infrared sensor, and / or any type of sensor capable of detecting a surface and any detectable frequency in the electromagnetic spectrum (i.e., ionizing radiation, visible light, microwaves, radio waves). A slider 104 or other interface tool may be utilized to browse the available designs 102.

[0015] Referring to FIG. 2 , once a design is selected, a mural design 200 version of the design may be utilized for more detailed viewing. The mural design 200 may be represented as a collection of mural pieces 206. In this embodiment, the mural design 200 may be a reduced-resolution representation. For example, one Post-it note may represent one pixel. This may further be represented as placement indicators 204, such as numbers. The placement indicators 204 may be any suitable type of placement indicator 204 recognizable to a user and may be utilized / overlaid on the mural pieces 206. A piece legend 208 may be utilized to indicate to the user the types of colors utilized and how the placement indicators 204 correspond to mural pieces 206 of different colors. In some embodiments, the mural pieces 206 may be represented with multiple colors and / or pixel values. Mural interface options 210 may include flipping and / or rotating the mural design 200 and modifying the mural design 200.

[0016] Referring to FIG. 3 , a user 300 is shown creating a mural design using augmented reality hardware 302. In this embodiment, a virtual surface 304 is overlaid on a physical surface 306, and a direction 308 of the virtual surface is displayed to provide continuous directional guidance to the user 300. The user 300 can create the virtual surface 304 by, but not limited to, identifying a point on the physical surface 306 (e.g., designated with a finger or gaze) and dragging the finger or gaze within the augmented reality to identify the diagonal corner of the virtual surface 304. In this manner, the virtual surface 304 remains overlaid on the physical surface 306 as sensors in the augmented reality hardware 302 update the position of the physical surface 306 based on the user's 300 movement relative to the physical surface 306. While the physical surface 306 is shown here as a flat, vertical surface, such as a wall, any suitable surface may be used. Examples include a floor, a ceiling, a sloped, flat wall, a curved surface, the intersection of two surfaces, etc. The virtual surface 304 may be divided into virtual pieces 312. In this embodiment, the virtual pieces 312 are arranged as a square or rectangular grid, but in other embodiments, the virtual pieces 312 may take any suitable shape. For example, the boundaries of the virtual pieces 312 may form triangles, diamonds, circles / ellipses, etc., with or without gaps between the boundaries of the various virtual pieces 312.

[0017] One or more placement indicators 310 may be used to indicate to the user 300 where to place a physical object, such as a particular color or type of Post-it note. In the non-limiting example shown in this figure, some or all of the virtual pieces 312 may correspond to the design shown in FIG. 1 and / or the mural shown in FIG. 2. In this manner, each pixel value of the mural shown in FIG. 2 may correspond to a virtual piece 312. In this non-limiting example, the mural shown in FIG. 2 is represented by three color values, with dark gray values ​​represented by a placement indicator 310 displayed as "3," medium gray values ​​represented by a placement indicator 310 displayed as "1," and light gray values ​​represented by a placement indicator 310 displayed as "2." While no background color is used in this embodiment, one or more background colors may be utilized and may be represented by one or more placement indicators 310. In some embodiments, the placement indicators 310 may correspond to the color, color pattern, or shape of the physical object 314 and / or the virtual piece 312.

[0018] The way the mural is displayed on the virtual surface 304 may change based on various factors, such as distance and / or time. As a non-limiting example, when the augmented reality distance between the user 300 and the virtual surface 304 is large, the virtual pieces 312 on the virtual surface 304 may display their respective colors. However, as the user 300 approaches the virtual surface 304, some or all of the placement indicators 310 may come into view, and the color values ​​may gradually fade or disappear completely. Or, the reverse may occur. In some embodiments, the content of some or all of the virtual pieces 312 may change over time. For example, a dynamic change may occur, such as alternating between displaying color values ​​and placement indicators 310.

[0019] As a non-limiting example, user 300 may manipulate and place physical object 314 on the virtual piece. Continuing with this non-limiting example, user 300 places a dark Post-it note on virtual piece 312 labeled "3," which corresponds to the dark Post-it note. In some embodiments, a virtual version (e.g., a "digital twin") of physical object 314 may be displayed in augmented reality to assist user 300. In some embodiments, as user 300 moves physical object 314 near virtual piece 312, the digital twin may "snap" to virtual piece 312, assisting user 300 in more accurately placing physical object 314 on physical surface 306. Additionally, in some embodiments, design interface 316 may be positioned (or "floating") near virtual surface 304, allowing user 300 easy access to the functionality of design interface 316 described with respect to FIG. 2.

[0020] Referring to Figure 4, a flowchart for creating a mural using augmented reality in accordance with an embodiment is shown. In block 400, a physical surface is detected. As described herein, a sensor or other image and / or object detection device may be used as part of suitable hardware, such as an augmented reality wearable, to detect the physical surface, such as a wall. In block 402, a virtual surface corresponding to the physical surface is specified. For example, a user may be wearing augmented reality hardware and be able to draw the boundary of the virtual surface on the physical surface. This allows the virtual surface to change in a corresponding manner depending on the user's proximity to and viewpoint of the wall.

[0021] In block 404, an image (or design) may be selected, received, created, or updated. For example, a user may upload a previously created or saved image. In some embodiments, the user may be asked to approve or preview a lower-resolution version of the image, which may correspond to a mural version. In block 406, the image is segmented and each pixel value is set to be associated with, but not limited to, a particular color of Post-it note. Note that blocks 404 and / or 406 may be executed independently of when blocks 400 and / or 402 are executed.

[0022] At block 408, placement indicators are displayed on the virtual surface. As described herein, each placement indicator indicates a corresponding location on the physical surface and indicates a location in augmented reality where the user should place a physical object (e.g., a Post-it Note). At block 410, the placement of the physical objects is detected. For example, the same sensor or different image detection / object detection hardware can be used to visually detect and track the physical objects placed by the user on the physical surface. At block 412, the placements of the multiple physical objects are compared to determine whether they are correctly placed according to their respective placement indicators. As a non-limiting example, the position of each physical object is compared to the placement indicators and the corresponding virtual piece on the virtual surface to determine whether it is properly placed. In this manner, the virtual piece and / or placement indicator are superimposed on the physical object, visually confirming the accuracy of the physical object's placement to the user via augmented reality. At block 414, an indication is output as to whether the physical object was correctly or incorrectly placed on the placement surface. As a non-limiting example, if one or more sides of a physical object (or its virtual twin) do not fall within the boundaries of the virtual piece, a virtual piece boundary violation occurs, and the user may be notified via a graphical alert (e.g., blinking the boundary, turning the boundary red, etc.), an audio alert, or other means. In some embodiments, the user may be notified when the physical object (or its virtual twin) is correctly placed within the boundaries of the virtual piece. In this embodiment, the color of the physical object is compared to a color value associated with the placement indicator, and if the color matches with placement within the boundaries of the virtual piece, the user is notified of correct placement. In other embodiments, correct placement may be determined solely based on whether the physical object is placed within the boundaries of the virtual piece, without considering the color match. In some embodiments, the user may be notified when all physical objects are correctly placed within the virtual piece on the virtual surface according to their respective placement indicators.

[0023] Referring to FIG. 5 , a block diagram illustrates computing hardware (e.g., an exemplary computing device 500) on which embodiments of the present disclosure can be implemented. The computing device 500 described herein is merely one example of a suitable computing device and is not intended to limit the scope of the present embodiments. Furthermore, any element or elements shown and described with respect to the computing device 500 should not be construed as required or to create any kind of dependency. In various embodiments, the computing device 500 may include, but is not limited to, a desktop, laptop, server, client, tablet, smartphone, computing cloud, or any type of device capable of utilizing data. In this embodiment, the computing device 500 has at least one processor 502 and memory, including non-volatile memory 508 and / or volatile memory 510. The computing device 500 may include one or more displays, display hardware, and / or output devices 504. Examples of output devices 504 include AR / VR / MR / XR hardware, monitors, speakers, headphones, projectors, wearable displays, holographic displays, printers, etc. Additionally, the output device(s) 504 may include a display and / or speaker, energy emitting devices (radio waves, microwaves, infrared, visible light, ultraviolet, x-rays, gamma rays, etc.), electronic output devices (Wi-Fi, radar, lasers, etc.), audio output devices of any frequency, etc.

[0024] The computing device 500 may further include one or more input devices 506. Examples of input devices 506 include, but are not limited to, a mouse, keyboard, disk / media drive, memory stick / thumb drive, memory card, pen, touch input device, biometric scanner, gaze and / or blink tracker, tracker, audio / auditory input device, motion detector, camera, scale, and any device capable of measuring motion data (e.g., accelerometer, GPS, magnetometer, gyroscope, etc.). The input devices 506 may also include devices for measuring biometric data (e.g., blood pressure, pulse, heart rate, sweat rate, body temperature, voice, facial recognition, movement / gesture tracking, gaze tracking, iris recognition or other eye identification, hand geometry, oxygen saturation, blood glucose level, fingerprint, DNA, dental records, weight, or other suitable biometric data). They may also include devices for capturing video / still image data and audio data, including audible and ultrasound. The input device 506 can be any type of device that receives data from other devices, visual and / or audio data obtained from the real world, object detection data, etc. The input device 506 corresponds to the sensors described herein (e.g., LIDAR sensors, time-of-flight (TOF) sensors, ultrasonic sensors, sonar, infrared sensors, etc.) used to detect surfaces / objects or any detectable frequency in the electromagnetic spectrum (i.e., ionizing radiation, visible light, microwaves, radio waves). Additionally, the input device 506 can include a camera (with or without audio recording). Examples include digital and / or analog cameras, still cameras, video cameras, thermal imaging cameras, infrared cameras, cameras with charge-coupled devices (CCDs), night vision cameras, 3D cameras, webcams, audio recorders, etc.

[0025] The computing device 500 typically includes non-volatile memory 508 (e.g., ROM, flash memory, etc.), volatile memory 510 (e.g., RAM, etc.), or a combination thereof. The network interface 512 can facilitate communication with other data sources, such as a database 518, via a network 514. The network 514 may include a wired connection, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), a cellular network, a satellite network, etc. Suitable LANs include wired Ethernet and / or wireless technologies (e.g., Wireless Fidelity (Wi-Fi)). Suitable PANs may include infrared data communications (IrDA), Bluetooth, wireless USB, Z-Wave, ZigBee, and / or other short-range communication protocols. Similarly, PANs may also include wired computer buses such as USB and FireWire. Suitable cellular networks include, but are not limited to, technologies such as LTE, WiMAX, UMTS, CDMA, and GSM. The network interface 512 may be communicatively coupled to any device capable of transmitting and receiving data over one or more networks 514. Accordingly, the network interface 512 may include a communications transceiver for transmitting and receiving wired or wireless communications. For example, the network interface 512 may include an antenna, a modem, a LAN port, a Wi-Fi card, a WiMax card, mobile communications hardware, near-field communications hardware, satellite communications hardware, and / or other wired or wireless communications hardware.

[0026] The computer-readable medium 516 includes one or more computer-readable media, each of which is non-transitory. The computer-readable medium may reside, for example, in the input device 506, the non-volatile memory 508, the volatile memory 510, or a combination thereof. A readable storage medium includes tangible media capable of storing instructions associated with or used by a device or system. Computer-readable media (also referred to herein as "non-transitory computer-readable media") include, by way of non-limiting example, RAM, ROM, cache memory, optical fiber, EPROM / flash memory, CD / DVD / BD-ROM, hard disk drive, solid-state storage, optical or magnetic storage, floppy disk, electrical connections having conductive wires, or combinations thereof. Non-transitory computer-readable media may include, for example, magnetic, optical, semiconductor, or electronic type systems or devices. In contrast, non-transitory computer-readable media does not include carrier waves and / or propagating signals in the form of optical, electromagnetic, or combinations thereof.

[0027] Computing device 500 may include one or more network interfaces 512 to facilitate communication with one or more remote devices. These remote devices may include, for example, client devices and / or server devices. Network interface 512 may also be described as a communications module, and these terms may be used interchangeably. Database 518 is shown accessible via network 514 and may reside in a server, cloud, or other configuration that supports remote access and storage of data.

[0028] As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include embodiments including plural referents unless the context clearly dictates otherwise. Also, as used in this specification and the appended claims, the term "or" is generally used in its sense including "and / or" unless the context clearly dictates otherwise.

[0029] While particular embodiments have been illustrated and described herein, it should be understood that various changes and modifications can be made without departing from the scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter are described herein, these aspects do not necessarily have to be utilized in combination. Accordingly, it is the intention of the appended claims to cover all such changes and modifications that are within the scope of the claimed subject matter.

Claims

1. One or more non-transitory computer-readable media encoded with instructions that, when executed, detecting, via a sensor, a physical surface within a line of sight of said sensor; providing a virtual surface corresponding to said physical surface; receiving an image from a source other than said sensor; Dividing the image into a plurality of parts; displaying placement indicators within the virtual surface, each indicating a location on the physical surface for placing a physical object corresponding to one of the portions of the image; One or more non-transitory computer-readable media that configure a computing device to:

2. The one or more non-transitory computer-readable media of claim 1 , further comprising instructions for detecting an arrangement of a plurality of said physical objects.

3. comparing the placement of the plurality of physical objects to confirm correct placement according to the placement indicator; outputting an indication of correct or incorrect placement of a physical object on the placement surface; 3. The one or more non-transitory computer-readable media of claim 2, further comprising instructions for:

4. The one or more non-transitory computer-readable media of claim 1 , wherein at least one of the physical objects has two or more colors.

5. The one or more non-transitory computer-readable media of claim 1 , wherein one of the physical objects is positioned according to a position indicator.

6. 6. The one or more non-transitory computer-readable media of claim 5, wherein the placement indicator corresponds to a color of the physical object, a pattern of colors of the physical object, or a shape of the physical object.

7. The one or more non-transitory computer-readable media of claim 6 , wherein each physical object has a single color, and the placement indicator is a number that corresponds to the color of the physical object.

8. The one or more non-transitory computer-readable media of claim 1 , wherein the virtual portions are separated by boundaries.

9. 10. The one or more non-transitory computer-readable media of claim 8, wherein two of the boundaries form a non-orthogonal angle.

10. The one or more non-transitory computer-readable media of claim 1 , wherein the physical surface has curvature or comprises the intersection of two planes.

11. displaying the image on the virtual surface based on the sensor's distance from the physical surface; transforming the image onto the virtual surface into the placement indicator based on a change in distance from the sensor to the physical surface; 10. The one or more non-transitory computer-readable media of claim 1, further comprising instructions for:

12. The one or more non-transitory computer-readable media of claim 1 , further comprising instructions for snapping a virtual representation of a physical object onto the virtual surface.

13. 10. The one or more non-transitory computer-readable media of claim 1, wherein providing the virtual surface further comprises an augmented reality display, wherein the virtual surface is superimposed on the physical surface as the sensor updates the position of the physical surface.

14. detecting, via a sensor, a physical surface within a line of sight of said sensor; providing a virtual surface corresponding to said physical surface; receiving an image from a source other than said sensor; Dividing the image into a plurality of parts; and displaying placement indicators within the virtual surface, each indicating a location on the physical surface for placing a physical object corresponding to one of a plurality of portions of the image.

15. The method of claim 14 , further comprising detecting an arrangement of a plurality of the physical objects.

16. comparing the placement of the plurality of physical objects to confirm correct placement according to the placement indicator; The method of claim 15 , further comprising outputting an indication of correct or incorrect placement of a physical object on the placement surface.

17. a sensor configured to detect a physical surface within a line of sight of the sensor; Memory and a processor coupled to the memory and the sensor; display hardware coupled to the processor and the memory; Including, The processor: providing a virtual surface corresponding to said physical surface; receiving an image from a source other than said sensor; Dividing the image into a plurality of parts; It is configured as follows: The display hardware includes: A computing device that displays within the virtual surface placement indicators that each indicate a location on the physical surface for placing a physical object corresponding to one of the portions of the image.

18. The computing device of claim 17 , wherein the processor is configured to detect an arrangement of a plurality of the physical objects.

19. the processor: comparing the placement of the plurality of physical objects to confirm correct placement according to the placement indicator; outputting an indication of correct or incorrect placement of a physical object on the placement surface; 20. The computing device of claim 18 configured to:

20. The computing device of claim 17 , wherein at least one of the physical objects has two or more colors.

Images