Interactive greeting cards

Abstract

An interactive greeting card and method of creating an interactive greeting card are disclosed. The method involves generating a series of two-dimensional representations or projections of a three-dimensional animation in a plane and printing, on a substrate, one of the two-dimensional representations and a mark. The mark, when recognized by a mobile computing device, causes the mobile computing device to transmit a message to a given URL. When received, instructions are transmitted to the mobile computing device to display on a screen of the mobile computing device a camera feed from the mobile computing device's camera overlaid by additional two-dimensional representations from the series of two- dimensional representations.

Description

INTERACTIVE GREETING CARDSFIELD

[0001] This disclosure relates to software facilitation augmented reality experiences, and more specifically, to systems and methods for creating greeting cards (or other tactile objects or display surfaces) that can be experienced in new ways through a display of a personal computing device being used to interact with the object.BACKGROUND

[0002] Traditionally, printed matter (whether a greeting card, letter, poster, menu, or other document) is limited in its experiential impact to the static appearance of text, images, or patterns on a paper surface. Barring some aspects, such as illusions in a Magic Eye poster that take advantage of quirks in human vision processing, there is no opportunity for interactivity or a dynamic experience that changes from the initial view of the printed matter.

[0003] In recent years, the ubiquity of mobile computing devices such as smartphones have opened up new avenues for printed matter. A QR code or other barcode can be printed on a surface, and when the QR code is scanned by the camera of a smartphone, a web browser of the smartphone will be directed to a URL encoded by the QR code. The web browser can then display content somehow related to the printed material on a webpage, or prompt a user to download software related to the printed material.

[0004] Despite the advances in technology related to smartphones’ cameras, screens, and processing power, there have not been similar transformative effects by using QR codes or similar codes in everyday situations, or to augment many art forms. Thus, there are advantages to developing novel systems for using technology to experience printed matter in new ways and experimenting with new forms of communication and artistic expression.BRIEF SUMMARY

[0005] In order to address the limitations of previous systems in this field, a novel method of creating an interactive greeting card is disclosed. The method comprises generating a series of two-dimensional representations or projections of a three-dimensional animation in a plane; printing, on a substrate, one of the two-dimensional representations and a mark that, when recognized by a mobile computing device, causes the mobile computing device to transmit a message to a given URL; receiving, at a server that is associated with the given URL, a request from the mobile computing device that was generated in response to recognizing the mark; andtransmitting, to the mobile computing device, instructions to display on a screen of the mobile computing device a camera feed from the mobile computing device’s camera overlaid by additional two-dimensional representations from the series of two-dimensional representations.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Other aspects, features and advantages will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings (provided solely for purposes of illustration without restricting the scope of any embodiment), of which:

[0007] FIG. 1 depicts an example greeting card being viewed through the camera of a smartphone or similar mobile computing device;

[0008] FIG. 2 depicts an example set of computing devices to facilitate a change in the mobile computing device’s functionality while viewing the greeting card as depicted in FIG. 1;

[0009] FIG. 3 depicts, in simplified form, a method of generating an interactive experience for the mobile computing device’s user;

[0010] FIG. 4 depicts an augmented reality interface after implementing the change in the mobile computing device’s functionality described in FIGS. 2 and 3;

[0011] FIGS. 5A and 5B depict one kind of augmented reality experience involving an illusion of three-dimensional objects around a viewer through a front-facing camera; and

[0012] FIGS. 6A and 6B depict an additional style of augmented reality animation with layering in planes in front of the camera.DETAILED DESCRIPTION

[0013] As described above, the primary intended use for this technology is in the creation and experience of interactive greeting cards. Nevertheless, none of the systems and methods described herein are limited to a particular purpose, material, size, or configuration of printed material. Without undue experimentation, the techniques described here would be equally feasible for interactive versions of business cards, books, catalogs, posters, billboards, currency, flyers, marketing materials, menus, playbills, tickets, and many more printed examples. Any surface, material, or object can be made to work with the techniques described here, so long as a computing device with a camera and a screen is capable of capturing a surface on the camera and rendering an augmented reality over the camera feed on the device’s screen.

[0014] FIG. 1 depicts an example greeting card being viewed through the camera of a smartphone or similar mobile computing device.

[0015] When a greeting card or other printed matter 100 is viewed through the back-facing camera of a typical smartphone or other mobile computing device 105, the screen of the smartphone displays the card 100 itself, in addition to an illustration 110 and a QR code or other mark 115.

[0016] Although in this particular illustration, the QR code 115 is on the front of the card along with the illustration 110, in other embodiments, the QR code may be located on the back of the card or on an envelope or other packaging associated with the card, so as not to risk interference with the aesthetic experience of the illustration 110. In still other embodiments, the QR code could be embedded or hidden within the illustration 110 itself. The embedding could be prominent as a visual cue that the viewer should train a camera on the illustration, or more subtle, such as hiding the QR code among other visual elements in such a way that the user is unlikely to notice the QR code, or perhaps while not “hidden”, simply at the edge of the illustration to be less prominent.

[0017] In many embodiments, every instance of a particular card design will have an identical QR code printed on them. In some embodiments, it may be preferable to have a unique QR code on every card, which is a variant of the QR codes that are present on each other instance of the card. This variation would allow for potential customization in viewer experiences at a later time, by causing the software to react differently to each QR code, and for those differences to be configured at a time after the card is printed or after the card is purchased.

[0018] In FIG. 1, a typical smartphone is illustrated as the mobile computing device 105, and this would be the preferred configuration. However, other types of devices might be used instead in alternative embodiments. A handheld gaming console, smartwatch, laptop computer, or other device might be able to perform the same functionality, so long as it has a working camera, a screen, and preferably an ability to connect to the Internet or a similar network.

[0019] In a typical prior art use of a mobile computing device, no additional functionality would occur by viewing a greeting card through the camera, or at most, if the greeting card did have a QR code, the QR code would trigger opening a web browser window to a particular website. However, when the proper background infrastructure is developed and the proper software is installed on the mobile computing device, a much richer experience can be offered to the recipient of a card through the mobile computing device. Examples of this experience are depicted in FIG. 4 and the figures that follow it. A description of the means of providing this experience is depicted in FIGS. 2 and 3 and the text below that accompanies them.

[0020] FIG. 2 depicts an example set of computing devices to facilitate a change in the mobile computing device’s functionality while viewing the greeting card as depicted in FIG. 1.

[0021] When the mobile computing device 105 picks up the QR code 115 from the card 100, a URL is resolved from the encoding of the QR code. The mobile computing device then sends a request via the Internet or another network 200 to a server 205 that hosts content associated with that URL. That server 205 preferably retrieves a particular set of software instructions and assets (such as image or sound files) from a database 210, the particular set being uniquely identified by the URL of the HTTP request, or in data fields that are part of the request. Those software instructions and assets are then transmitted back through the network to the mobile computing device 105.

[0022] It is conceivable that the mobile computing device 105 might already store instructions and assets for a variety of cards that the mobile computing device might be exposed to, and that communication over a network will be ultimately unnecessary. However, for maximum flexibility, it is preferred that there be a dynamic downloading at the time of the card’ s viewing, rather than requiring pre-installation of software before the card is viewed. The existing software might be a completely native app for the mobile computing device, a progressive web app or other app that acts as a wrapper or shell for a cross-device application, or any other form of software that is executable by the mobile computing device.

[0023] In one preferred embodiment, there can be a hybrid between all of the software being downloaded dynamically and all of the software being pre-installed. For example, it is possible that Instagram or a similar mobile app that has video editing functionality built in is already installed on the mobile computing device 105. If so, the software and assets downloaded to the mobile computing device may be described using a syntax that Instagram already recognizes and can implement, in part or in whole, as a video filter.

[0024] In some embodiments, the server 205 to which the URL points and the database 210 might be a single device (such that the server incorporates the database 210 on its local drive instead of communicating with a separate device in its vicinity) or a multitude of devices (such as an initial load balancer or content delivery network entry point, which then forwards a request to one of a number of devices, potentially very widely dispersed). The exact configuration is not so important as having an automated listener that can distinguish between multiple distinct requests, retrieve appropriate instructions and assets that have been pre-stored, and transmit those instructions and assets back to the device that requested them.

[0025] FIG. 3 depicts, in simplified form, a method of generating an interactive experience for the mobile computing device’s user.

[0026] Initially, a series of two-dimensional representations or projections of a three- dimensional animation in a plane are generated (Step 300). These representations may be, depending on the particular use case, hand drawn or computer-generated, as necessary. Examples of particular use cases suitable for an interactive greeting card are included beginning with FIG. 5A and following, as discussed further below.

[0027] These images are intended to form an animation that will be interpreted by a human viewer as a three-dimensional art piece. The art piece will dynamically change, causing portions of the art piece that were initially concealed to be revealed, and / or for portions of the art piece that were initially visible to be concealed. For example, the butterfly depicted in FIG. 1 is not intended to be a static image that is translated or rotated around the page to make a primitive facsimile of movement; it is intended that a conceptual model of both sides of each wing will be drawn and colored, so that the butterfly can flap its wings or turn around in an augmented reality display. The rendered image at any moment may be colored, shaded, sized, skewed, etc. such that it realistically tracks how a butterfly would appear as it flew around.

[0028] At the same time these images are generated, other artistic choices may be made, such as selecting music or sound effects to be played in a synchronized way with the animation. Other functionalities of a mobile computing device might also be invoked, such as causing a vibration, causing a change in screen brightness, causing a notification to appear on-screen or in a notifications region / status bar, or any other change in the appearance or experience of holding the mobile computing device.

[0029] Additionally, a separate set of images may be generated for integration with the first animation, this second set being incorporated into a different camera stream. For example, an animation might be generated to appear over the face or around the head or body of a person holding the mobile computing device, when viewed through a front-facing camera of the mobile computing device.

[0030] After these two-dimensional representations are generated, one of these representations is printed onto a card, along with a mark whose purpose is to direct a mobile computing device to a source of software (Step 305). As previously mentioned, this mark is preferably a QR code, but a variety of bar codes or other markings might be incorporated instead. For example, the mark might not be a traditional encoding at all, but rather text that can be picked up by optical character recognition (OCR).

[0031] The means by which the mark directs the mobile computing device is preferably a URL to be used in an HTTP or FTP request, but likewise, alternative methods may be used. For example, rather than the mark itself encoding the entire URL, the mark might only encode asubstring meant to be incorporated into a URL in a predictable manner — for example, rather than one card with a mark indicating “http: / / www.softwaresource.com / download / 12345” and another with a mark indicating “http: / / www.softwaresource.com / download / abcde”, the marks on the two cards might only be “12345” and “abcde”, with software executing on the mobile computing device adding that mark’s text to the base URL before sending the HTTP request.

[0032] At some point in time after the card has been created, a camera of a mobile computing device is directed towards the card so that the mark is captured by the camera (Step 310). The mark is interpreted as described above and resolved into a URL, which is used in an outgoing request from the mobile computing device (Step 315) using the Internet or another network connection between the mobile computing device and the server at that URL.

[0033] In response to sending the HTTP request or other request to the server at the URL and the server receiving the request, an automatic download of software will be triggered (Step 320). In one embodiment, this software may include a custom-designed Instagram filter that works with the publicly available Instagram mobile application to modify images and video. In other embodiments, this software may include a different, standalone app to be installed on the mobile computing device instead, or a plugin or filter configured to work with a mobile app different from Instagram.

[0034] When the software is executed, the mobile computing device will begin to display on its screen a video stream from the camera that had been used to capture the QR code (Step 325). As previously mentioned, the video may be accompanied by music, sounds, vibration, or other experiential qualities that the mobile device is capable of conveying.

[0035] Rendering of the animation may require some processing and scaling, skewing / shearing, or rotation of the animation, depending on the orientation of the card with respect to the camera. For example, if the card is turned upside-down while the animation is playing, it may be desired that the animation turn upside-down, too. If the card is brought closer to or further from the camera, the animation may need to be resized accordingly. If the card is rotated up, down, right, or left, the animation may need to be skewed or sheared so that it still appears to be on the surface of the card, or if the animation were intended to appear to “come out of the card” towards the viewer, the animation may need to be further modified so that it is coming out of the apparent plane of the card, even though that vector is no longer pointing at the viewer. To aid in this rescaling, an animation might be defined with respect to particular reference points, such as the corners of the card, the QR code or other mark on the card, the original illustration, subtle targeting reticles printed on the card, or other reference points.

[0036] As previously mentioned, it is possible for cards to have a unique QR code in every instance of the card, and for the rendered animation to act as a template that can be configured in different ways before the rendering of the animation begins. For example, a birthday card might default to displaying a message of “Happy birthday!” in text appearing to float above the card surface, but if the purchaser of the card visits a website associated with the QR code before giving the card to the ultimate viewer, that website might allow the purchaser to customize the text to any other series of characters. As a result, when the animation is actually rendered, it might have different text from the default and tailored to the intended recipient, such as “Happy birthday, [Name]! We love you! - Mom and Dad” or “Feliz cumpleanos!”. Other options might be configurable in particular experiences, such as selecting the color of rendered flowers and butterflies, or selecting the music that will accompany a particular animation. Ultimately, the designer of an experience may want to select many aspects of text, color, movement, change, sound, music, or other effects that can be arbitrarily changed by the purchaser before the viewing, or if not arbitrarily changed, toggled between multiple options or settings.

[0037] The software adding the animation over the camera stream will routinely check whether the animation has concluded (Step 330). In some embodiments, the animation tells a story or has a well defined beginning and end; if so, conclusion of the animation may cause the augmented display to end (Step 340). In other embodiments, the animation may be intended to loop smoothly and will simply return to the beginning of the loop rather than truly ending.

[0038] If the animation is still ongoing, the software will check for changed circumstances with respect to the card, the camera, or other aspects of the viewing experience (Step 335). For example, if the card has been moved or rotated, future frames of the animation may need a similar transformation. If the card has been moved out of the camera’s line of sight, the software may have to determine whether to terminate the animation prematurely, or to modify it (such as by continuing to play music, but stopping the augmentation of the display until the card is returned into view). If the viewer has tapped a button to activate the front-facing camera instead of the rear-facing camera, a first animation for augmenting the card may need to be replaced by a second animation for augmenting the viewer. If the viewer taps the button to activate the rear-facing camera again, the original animation will be restored. After each moment of the experience, the appropriately updated video and other elements will be added to the video stream displaying on the mobile computing device’s screen (back to Step 325).

[0039] FIG. 4 depicts an augmented reality interface after implementing the change in the mobile computing device’s functionality described in FIG. 2.

[0040] Now, in contrast to the static card depicted in FIG. 1, the same card is rendered on the screen of the mobile computing device 105 with a constantly changing animation in front of it. The butterfly is now in motion, and appears to flap its wings, fly up from the surface of the card, and even to move beyond the edges of the card, into the air / background.

[0041] Further, the video stream of the card is modified by removing the QR code 115 and the original illustration 110 in real time by filtering them out and replacing them with the generic texture of the rest of the card. The software rendering the video stream may be configured to remove, cover up, add, or modify any number of pre-determined elements at all times or if they happen to come into the line of sight of the camera. For example, as depicted in the next figures, some elements may be specifically tied to facial recognition and appear on the augmented video stream only when a face is present on camera.

[0042] FIGS. 5A and 5B depict one kind of augmented reality experience involving an illusion of three-dimensional objects around a viewer through a front-facing camera.

[0043] In FIG. 5 A, while viewing the card from FIGS. 1 and 4 through the rear-facing camera, the viewer chooses to tap the camera switching icon. In response, as depicted in FIG. 5B, the original animation of a butterfly with a specific location respective to the card disappears. Instead, the viewer sees himself through the front-facing camera, now with one or more representations of the same butterfly model fluttering around him. If there was musical accompaniment, the accompaniment continues unchanged whether the camera is front-facing or rear-facing. The front-facing augmented reality need not incorporate views of the same model as the rear-facing version; there could be different butterflies, or flowers, or other complementary animations that fit the theme of the original rear-facing animation.

[0044] FIGS. 6A and 6B depict an additional style of augmented reality animation with layering in planes in front of the camera.

[0045] In FIG. 6A, a card may be provided to a viewer that appears to only have a closed curtain 600 printed on the card. Although the card does not itself in any way indicate the presence of a vanity 605 or other object behind the curtain, the curtain is intended to be part of an artistic experience involving the vanity. When the animation begins to play, the original illustration of the closed curtain is concealed in the video stream, and replaced with an animation of the curtain opening to reveal the vanity behind it. The animation may thus include multiple layers in the Z-dimension and use changes in the X- or Y-dimensions to reveal new material that was initially concealed.

[0046] One can imagine all manner of interactive experiences that are responsive to viewer movement, card movement, camera control, or other inputs from the mobile computing deviceor environment, and that respond to these changes by updating the generated animation as if the camera of the mobile computing device is actually capturing a changing, dynamic artistic scene.

[0047] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0048] The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A method of creating an interactive greeting card, comprising: generating a series of two-dimensional representations or projections of a three- dimensional animation in a plane; printing, on a substrate, one of the two-dimensional representations and a mark that, when recognized by a mobile computing device, causes the mobile computing device to transmit a message to a given URL; receiving, at a server that is associated with the given URL, a request from the mobile computing device that was generated in response to recognizing the mark; and transmitting, to the mobile computing device, instructions to display on a screen of the mobile computing device a substrate-facing camera feed from the mobile computing device’s camera overlaid by additional two-dimensional representations from the series of two- dimensional representations.

2. The method of claim 1 further comprising: providing software to the mobile computing device that is installed on the mobile device prior to the transmitting.

3. The method of claim 1 further comprising: transmitting, to the mobile computing device, instructions that cause the mobile computing device to output, in addition to the additional two-dimensional representations from the series of two-dimensional representations, one or more of music, a sound or a vibration.

4. The method of claim 1, further comprising: defining reference points on the interactive greeting card to aid in rescaling of the additional two-dimensional representations from the series of two-dimensional representations.

5. The method of claim 4, wherein the reference points include one or more of: boundaries of the substrate; a QR code; targeting reticles printed on the substrate; the mark; or the one of the two-dimensional representations printed on the substrate.

5. The method of claim 1 further comprising: providing a website associated with the mark that will allow a purchaser of the interactive greeting card to customize characters that will be rendered on the interactive greeting card in conjunction with the additional two-dimensional representations.

6. The method of claim 1 further comprising: providing a website associated with the mark that will allow a purchaser of the interactive greeting card to customize one or more colors that will be rendered on the interactive greeting card in conjunction with the additional two-dimensional representations.

7. The method of claim 1 further comprising: providing a website associated with the mark that will allow a purchaser of the interactive greeting card to customize sound that will be provided in conjunction with the additional two-dimensional representations.

8. The method of claim 1 wherein the additional two-dimensional representations from the series of two-dimensional representations are displayed as a repeating loop animation.

9. The method of claim 1 wherein the additional two-dimensional representations from the series of two-dimensional representations are displayed as an animation having a defined beginning and end.

10. The method of claim 1 wherein, in response to a viewer activating a viewer-facing camera on the mobile computing device, the method comprises: augmenting images obtained by the mobile computing device via the viewer-facing camera with an alternative animation.

11. The method of claim 1 wherein, in response to a viewer deactivating the viewer-facing camera on the mobile computing device and reactivating a camera facing the substrate, the method comprises: re-displaying the substrate-facing camera feed overlaid by the additional two- dimensional representations from the series of two-dimensional representations.

12. The method of claim 1 further comprising: filtering out at least a portion of the one of the two-dimensional representations.

13. A method comprising: pointing a camera of a mobile computing device at a mark printed on a substrate defining a part of an interactive greeting card; andreceiving, at the mobile computing device, instructions to display on a screen of the mobile computing device a substrate-facing camera feed of at least some of a series of two- dimensional representations or projections of a three-dimensional animation overlaying one of the series of two-dimensional representations that is printed on a surface of the substrate.

14. The method of claim 13 further comprising, in response to a selection of a different camera, located on a viewer side of the mobile computing device, augmenting images obtained by the mobile computing device via the viewer-facing camera with an alternative animation.

15. The method of claim 14 further comprising, in response to a new selection deactivating the viewer-facing camera on the mobile computing device and reactivating the camera facing the substrate, the method comprises: re-displaying the substrate-facing camera feed overlaid by the additional two- dimensional representations from the series of two-dimensional representations.

16. An interactive greeting card comprising: a substrate, having printed thereon, one of two-dimensional representation a series of previously generated two-dimensional representations or projections of a three-dimensional animation in a plane, and a mark that, when recognized by a mobile computing device, will cause the mobile computing device to transmit a message to a given URL; and receive instructions to display on a screen of the mobile computing device a substratefacing camera feed from the mobile computing device’s camera overlaid by additional two- dimensional representations from the series of two-dimensional representations.

17. The interactive greeting card of claim 16, wherein the mark comprises a QR code.

18. The interactive greeting card of claim 17, wherein the QR code is on a surface of the substrate containing the one of the two-dimensional representations.

19. The interactive greeting card of claim 17, wherein the QR code is on a surface of the substrate other than the surface containing the one of the two-dimensional representations.

Images