A messaging app that interfaces with one or more extension apps

Abstract

The present invention relates to a message application that interfaces with one or more extension applications. In one embodiment, the present invention provides a message system, which includes a message application configured to create content displayed in a view hosted by the message application, and one or more extension applications. The message application can launch one or more extension applications and transfer content between each extension application and the message application through inter-process communication.

Description

[0001] This application is a divisional application of invention patent application No. 201710435784.4, filed on June 12, 2017, and entitled “Message application docking with one or more extension applications”. Technical Field

[0002]

[0013] Embodiments described herein relate to a messaging system, such as a text messaging system, on a cellular telephone or other communications device or data processing system. Background Art

[0003] The use of text messaging systems began many years ago. For example, in the 1990s, before smartphones were available, wireless cellular phone operators such as Verizon or AT&T allowed the text messages through short message services (SMS) for cellular phones. Typically, the amount of data transmitted is limited by the rules set by the operator. Recently, with the increase in the use of smart phones (e.g., iPhones) and tablet computers (e.g., iPads), text messaging systems have developed the ability to send images such as photos or emoticons. In addition, messaging systems such as iMessage from Apple Inc. (Cupertino, California) allow users to send and receive text and images through a "public" network, which includes a "public" WiFi access point and the Internet (except for the use of a wireless operator's dedicated cellular phone network), and messaging systems such as iMessage can be seamlessly switched between the use of public networks and private networks according to, for example, the availability of WiFi access points or the compatibility of other user devices (which may be incompatible with iMessage). Summary of the invention

[0004] One aspect of the embodiments described herein relates to a messaging system, which includes an instant messaging application configured to create content displayed in a view hosted by an instant messaging application and one or more extension applications on a client device. The instant messaging application can launch one or more extension applications, and in one embodiment, the content is transmitted between each extension application and the instant messaging application through inter-process communication. In one embodiment, a method may include: receiving a message and associated metadata from a second device through a first instant messaging application on a first device, the message including content created by a second extension application operating with a second instant messaging application on the second device. In one embodiment, the first instant messaging application and the second instant messaging application are each configured to transmit short message service (SMS) text messages and other content, and display those text messages in a message record. The method may also include displaying the content in the message record in a user interface view of the instant messaging application, and transmitting the content from the first instant messaging application to the first extension application identified by the application identifier in the metadata, the content being transmitted from the first instant messaging application executed in the first process to the first extension application executed in a second process different from the first process through inter-process communication (IPC). The method also includes displaying the user interface of the first extension application within the user interface of the first instant messaging application.

[0005] In one embodiment, the user interface of the first extension application is configured to display content such as content created by the second extension application on the second device, and receive user input to modify the content. In one embodiment, the user interface of each extension application in the extension application can replace the on-screen keyboard of the instant messaging application when it is displayed. In one embodiment, the first extension application can modify the content and transmit the modified content to the first instant messaging application through IPC for transmission to the second instant messaging application for delivery to the second extension application on the second device. In one embodiment, the first extension application and the second extension application are two instances of the same extension application and are each identified by the same application identifier, which can be provided by an application market or other service that provides downloads of extension applications.

[0006] In one embodiment, the first extension app can be configured to receive an obfuscated identifier of a user of another device (second device), wherein the obfuscated identifier can be configured to be unique to the first extension app relative to all other extension apps on the first device. The obfuscated identifier allows each extension app to identify the user in a collaborative environment, such as an environment where multiple users are trying to schedule a meeting or reserve a table at a restaurant, etc.

[0007] In one embodiment, the first extension application can change the view hosted by the instant messaging application. For example, in one embodiment, the first extension application can call the first instant messaging application through an application programming interface (API) to request to change the view of the extension application within the first instant messaging application. In one embodiment, the change of the view of the extension application is a switch between a compact view and an expanded view. In one embodiment, the user interface of the extension application in the compact view can be displayed in a screen area where the on-screen keyboard of the first instant messaging application is displayed when the message record of the first instant messaging application remains visible. In another embodiment, the compact view of the extension application can be displayed as an overlay on the on-screen keyboard. In one embodiment, the view of the extension application in the extended view is displayed in a screen area where both the on-screen keyboard and the message record are displayed so that the message record is not visible and the on-screen keyboard of the first instant messaging application is also not visible.

[0008] In one embodiment, the metadata received from the second device may include a uniform resource locator (URL) or other type or form of resource locator (e.g., uniform resource identifier (URI), XML, etc.) transmitted to the first extension application by the first instant messaging application through IP, and data, wherein the resource locator and data maintain session information between the second extension application on the second device and the first extension application on the first device. In one embodiment, the extension applications communicate through the corresponding instant messaging application hosting its view on each device. In one embodiment, each extension application can be configured to modify at least one of the resource locator or data and transmit the modified resource locator or data to the other extension application through the two instant messaging applications. In one embodiment, the change of the resource locator or data can be implemented in the conversation, where the same conversation bubble in the message record is displayed over time because the information in the bubble changes with the user interaction on both sides of the communication. In one embodiment, each extension application can also receive a callback from its corresponding instant messaging application to indicate whether the other extension application has received the modified content such as the modified resource locator or data.

[0009] Another aspect of the embodiments described herein relates to how an extension application is launched in response to a selection of a message bubble in a message history presented by an instant messaging application. A method according to this aspect may include: receiving a message and metadata from a second device by a first instant messaging application on a first device, the message including content created by a second extension application operating with the second instant messaging application on the second device. The method also includes displaying content in an information container, such as a message bubble, within the message history of the instant messaging application, and then receiving a selection of the message container, such as a tap of a user's finger on the message bubble. If the first extension application is installed on the first device in response to selecting the message bubble, the method may also include launching the first extension application, wherein the first extension application is identified by an application identifier in the metadata received from the second device for launching. In one embodiment, the method also includes displaying a user interface of the first extension application within the user interface of the first instant messaging application after launching. In one embodiment, the message container may be a message bubble specified by a bubble identifier in the metadata, and the content may be associated with the bubble identifier so that the content created by the second extension application appears within the message bubble with the bubble identifier. In one embodiment, the first instant messaging application and the second instant messaging application are each configured to transmit SMS text messages and other content, and display the text message in the message bubble in the message record. In one embodiment, the first instant messaging application and the first extension application are configured to communicate via IPC, and the first instant messaging application is executed in a first sandbox process, and the first extension application is executed in a second sandbox process different from the first sandbox process. In one embodiment, before the message bubble is selected, the content is displayed by the first instant messaging application without launching or executing the first extension application; in other words, in one embodiment, the first extension application does not need to be executed in order for the content to be displayed in the message bubble by the first instant messaging application. In one embodiment, an API may exist between the first instant messaging application and the first extension application to allow communication through calls utilizing the API.

[0010] In one embodiment, the content received by the first instant messaging application is received in encrypted form and decrypted by the first instant messaging application, and the decrypted form is transmitted to the first extension application via IPC.

[0011] In one embodiment, if the first extension application is not installed, the first instant messaging application may provide for downloading and installing the first extension application. The first instant messaging application may maintain a registry (e.g., a list) of installed extension applications, and the registry may be used to determine whether the extension application is installed when, for example, the user taps or selects a message bubble with content created by a specific extension. If, in response to the selection of the message bubble, the first instant messaging application determines that the specific extension application required to process the content (as specified by the application identifier provided with the message) is not installed, the first instant messaging application may present a notification to the user providing the user with an option to select which option will cause the device to download and install the extension application. In one embodiment, the download and installation of the extension application may be performed in the background while the first instant messaging application keeps the foreground application on the device.

[0012] Another aspect of the embodiments described herein relates to a method for providing backward compatibility, which may occur when a device communicates with an old device or a device with an old messaging system that is not fully compatible with a new messaging system. In one embodiment, the method described in this aspect may include: creating content displayed in a view hosted by a first instant messaging application on a first device through an extension application executed in a first process, wherein the first instant messaging application is executed in a second process different from the first process, and in response to the selection of a send command, the content is transmitted from the extension application to the first instant messaging application through inter-process communication so that the content is sent to the second device. The method may also include determining, by the first instant messaging application, from data about the second device that the second instant messaging application on the second device is incompatible with the extension application, and then sending alternative content to the second instant messaging application through the first instant messaging application. In one embodiment, the alternative content may include one or more of the following: (a) an image in a standard format; or (b) a resource locator that can be retrieved = a web page on the second device.

[0013] Another aspect of the embodiments described herein relates to an application market or similar service that can provide a browsable view of an extension application in one embodiment, which can be downloaded and installed from the application market or similar service. In one embodiment, the application can be obtained for free or can be purchased and then downloaded and installed on the user device. In one embodiment, the download can occur in response to the selection of a message bubble, which contains content created by the extension application that has not been installed on the receiving device. In one embodiment, the method may include the following operations: receiving a message and associated metadata from a second device through a first instant messaging application (application) on a first device, wherein the message includes content created by a second extension application operating with a second instant messaging application on the second device, wherein the associated metadata includes an application identifier associated with the second extension application; displaying the content in the message record in a user interface view of the first instant messaging application; determining whether the first extension application identified by the application identifier is installed for use with the first instant messaging application; displaying a notification including an option to cause the first device to download and install the first extension application to the user, wherein the notification is displayed in response to determining that the first extension application is not installed for use with the first instant messaging application; and downloading and installing the first extension application in response to the selection of the option. In one embodiment, in response to a user selection of content in a message record (e.g., a user tapping content in a message bubble of the message record), the first device determines that a first extension application identified by an application identifier is not installed. In one embodiment, the first extension application is downloaded from a service that provides a plurality of messaging extension applications for download, and the application identifier is provided by the service. In one embodiment, the service includes one or more server systems having a browsable directory of messaging extension applications that can be downloaded from the service. In one embodiment, the download and installation of the first extension application can occur while the first instant messaging application remains a foreground application. In one embodiment, the method may further include: adding an icon representing the first extension application to a browsable view of installed extension applications, the browsable view being displayed by the first instant messaging application, and the addition of the icon may occur after the first extension application is installed.

[0014] In one embodiment, the installed first extension application can communicate with a first instant messaging application through IPC, where the first instant messaging application is executed in a first process different from a second process executing the first extension application.

[0015] Another aspect of the embodiments described herein relates to an extension application that is an image creation application, such as an application that can create stickers and can modify the appearance of those stickers and can allow those stickers to be placed on one or more message bubbles in a message record provided by an instant messaging application. In one embodiment of this aspect, the method may include the following operations: receiving a selection to create a sticker image within a user interface view of a sticker extension application (application) hosted by an instant messaging application; receiving one or more user inputs for specifying changes to the selected sticker image, the one or more user inputs defining image metadata; transmitting the sticker image and image metadata to the first instant messaging application through inter-process communication between the sticker extension application and the first instant messaging application; uploading the sticker image, message, and image metadata to one or more message servers for delivering the message, sticker image, and image metadata to a receiving device through the first instant messaging application; receiving and storing a token from the one or more message servers through the first instant messaging application, the token representing the sticker image; in response to a subsequent request for sending the sticker image to the receiving device or to other receiving devices, sending the token to one or more message servers through the first instant messaging application without sending the sticker image at the same time. In one embodiment, the method may also include displaying a set of sticker images through a sticker extension application in a view hosted by the first instant messaging application, each sticker image can be selected by the user to be associated with a message in a message bubble in a message record view provided by the first instant messaging application. In one embodiment, the user associates the sticker image with the message bubble by dragging the sticker image onto the message bubble. In one embodiment, the image metadata may include one or more of the following: (a) scaling data that changes the size of the sticker image; (b) rotation data that changes the rotation or orientation of the sticker image on the message bubble; (c) position data of the sticker image on the message or message bubble; or other image modification data (e.g., transparency or color difference, etc.). In one embodiment, the data volume of the sticker image is constrained by the messaging system to remain below a predetermined limit. The first instant messaging application displays the sticker image on the message in the message record within the view of the first instant messaging application, and the size, rotation and position of the sticker image on the message can be specified by the image metadata. In one embodiment, when the message (with the sticker image) is sent to multiple receiving devices, the first instant messaging application can encrypt the sticker image of each receiving device to generate multiple encrypted sticker images.For example, in one embodiment, the public key of each receiving device may be used to encrypt the sticker image (or to encrypt the key used to encrypt the sticker image), and each different receiving device may have a different public key. In one embodiment, an asymmetric encryption algorithm such as a public key / private key encryption algorithm may be used to encrypt the sticker images (or to encrypt the keys for those images). The method may also include receiving a time-to-live (TTL) value for the token from one or more messaging servers, the TTL value being refreshed at the first messaging application and at the one or more messaging servers in response to subsequent requests for sending sticker images corresponding to the token. In one embodiment, the method may also include determining, by the first messaging application, from data about the receiving device that the instant messaging application on the receiving device is incompatible with the sticker application, and when it is incompatible with the sticker application, sending alternative content to the receiving device by the first messaging application.

[0016] Another aspect of the embodiments described herein relates to a method for reproducing a final sticker image using image metadata on a receiving device. In one embodiment, the method described in accordance with this aspect includes the following operations: receiving a message and a token, and associated metadata (which may include image metadata) from a second device via a first instant messaging application on a first device, the token relating to a sticker image created by a second sticker extension application operating with a second instant messaging application on the second device; sending the token to one or more messaging servers via the first instant messaging application to obtain a sticker image; receiving the sticker image via the first instant messaging application to respond to sending the token and generating a final image by applying the image metadata to the sticker image; and displaying the final image at or near a message bubble containing the message via the first instant messaging application. In one embodiment, the first instant messaging application is executed in a first process, and the first sticker extension application is executed in a second process different from the first process, and wherein the message bubble is identified by a bubble identifier. In one embodiment, the first instant messaging application receives the sticker image in encrypted form and decrypts the sticker image, and provides the decrypted sticker image to the first sticker extension application via IPC. In one embodiment, the image metadata includes one or more of the following data: (a) scaling data that changes the size of the sticker image; (b) rotation data that changes the rotation of the image sticker; or (c) other image modification data. In one embodiment, the first instant messaging application uses a hash of the sticker image to determine whether the first instant messaging application already has a copy of the sticker image before sending the token; in one embodiment, the hash may be included in the associated metadata or image metadata. In one embodiment, the method may also include automatically launching the first sticker extension application in response to receiving the message, wherein the launching occurs without user action or intervention.

[0017] Another aspect of the embodiments described herein relates to a system and method for operating one or more message servers in a message system. In one embodiment, the method described in accordance with this aspect includes the following operations: receiving multiple encrypted sticker images for delivery to multiple recipients from a first instant messaging application, each of the multiple encrypted sticker images representing the same sticker image when decrypted; creating multiple tokens and transmitting them to the first instant messaging application, each of the multiple tokens being assigned to one of the multiple recipients; creating a time to live (TTL) value for each encrypted sticker image in the encrypted sticker image and transmitting at least one of the TTL values ​​to the first instant messaging application; storing multiple encrypted sticker images and multiple tokens and TTL values; and receiving one of the multiple tokens from the first instant messaging application in response to the first instant messaging application sending the same sticker image to one of the multiple recipients again and refreshing the TTL value corresponding to the received token. In one embodiment, the method may also include sending the encrypted sticker image to a second instant messaging application on a receiving device in response to the second instant messaging application sending one of the multiple tokens, the sent token corresponding to the sent encrypted sticker. In one embodiment, one or more message servers that can perform the method may also receive multiple encrypted messages, one encrypted message for each of multiple recipients. In one embodiment, the one or more message servers may store the multiple encrypted sticker images, multiple tokens, and a set of TTL values ​​in one or more databases. In one embodiment, the method may also include receiving metadata associated with the same sticker image from a first instant messaging application and transmitting the metadata to a second instant messaging application. In one embodiment, the metadata may be at least one of the following: (a) a sticker extension application identifier provided by a service from which the extension application can be downloaded; or (b) image metadata used by a sticker extension application that operates with a second instant messaging application to create a modified sticker image.

[0018] While the foregoing examples describe the use of TTL values ​​for stickers and sticker images, in one or more embodiments described herein, TTL values ​​may be used for other objects and content that are sent or stored or received in a messaging system.

[0019] In one or more embodiments, an indicator of the sticker's source can be displayed in the message transcript. This can allow the recipient of the sticker to see who sent the sticker. In one embodiment, the indicator can be a color that highlights or surrounds the sticker in the message transcript, or alternatively the recipient can select a sticker that causes a sticker detail window to be displayed, which displays detailed information about the sticker, including the original artist or developer (or other source) of the sticker, and the sender who can modify the sticker. In one embodiment, the source or other properties of the sticker displayed in the message transcript can be automatically displayed in the message transcript for a short period of time when the receiving device first receives the sticker, and then no information about the source or properties of the sticker is displayed.

[0020] In one embodiment, a method for operating in conjunction with an instant messaging application may launch an extension application in different ways depending on the type of the extension application or according to a policy regarding the extension application. In one embodiment, a method may include the following operations: in response to receiving a message containing content created by the extension application, determining the type of the extension application by the instant messaging application; if the extension application is an extension application of the first type or the policy for the extension application allows such launch (for example, the extension application requests immediate launch and the instant messaging application authorizes the request for such extension application, or the extension application is of a predetermined type or its content is of a predetermined type), then automatically launching the extension application in response to receiving the message; if the extension application is an extension application of the second type (or the policy determines to postpone the launch), then postponing the launch of the extension application until a selection of a message bubble displayed in a message record is received, the message record being displayed by the instant messaging application. In one embodiment, the selection is received from a user who selects the message bubble in the message record. In one embodiment, the extension application specifies whether it operates as an extension application of the first type or as an extension application of the second type, and the instant messaging application may apply one or more policies to allow automatic launch or postpone the launch of the extension application. In one embodiment, an extension app of the first type is launched when received, but remains in the background (and, in one embodiment, may update the content in its message bubble while in the background) until the user selects the message bubble for displaying content from the extension app, causing the extension app to be displayed in the foreground (e.g., selected to be displayed in a compact view mode instead of an on-screen keyboard).

[0021] Another aspect of the embodiments described herein relates to using breadcrumbs in a conversation between two extension applications on two devices. In one embodiment, a method for using breadcrumbs may include the following operations: displaying a first message bubble containing first content created by a first extension application or associated with the first content in a message history, the first message bubble having a session identifier maintained for use by a first instant messaging application; receiving second content of a conversation between a first extension application and a second extension application occurring on a second device at a first instant messaging application of the first device, wherein the conversation is identified by the session identifier; converting the first content into breadcrumbs and displaying a second message bubble containing second content or associated with the second content, wherein the first instant messaging application associates the session identifier with both the breadcrumbs and the second message bubble. In one embodiment, the first extension application provides a third content to be displayed as at least a portion of the breadcrumbs. In one embodiment, the second content is created by a second extension application on the second device and transmitted to the first device by a second instant messaging application on the second device. In one embodiment, the first instant messaging application converts the first content into breadcrumbs.

[0022] The methods and systems described herein may be implemented by data processing systems such as one or more smart phones, tablet computers, desktop computers, laptop computers, smart watches, wearable audio accessories, onboard computers, and other data processing systems and other consumer electronic devices. The methods and systems described herein may also be implemented by one or more data processing systems that execute executable computer program instructions stored in one or more non-transient machine-readable media, so that one or more data processing systems execute one or more methods described herein when executing program instructions. Therefore, the embodiments described herein may include methods, data processing systems, and non-transient machine-readable media.

[0023] The above summary does not include an exhaustive list of all embodiments of the present disclosure. All systems and methods can be practiced according to all suitable combinations of the various aspects and embodiments summarized above and those disclosed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0025] Figure 1A An example of a user interface of an instant messaging application on a communication device is shown.

[0026] Figure 1BAnother example of a user interface for an instant messaging application on a communication device is shown.

[0027] Figure 2 An example of a messaging system that employs one or more messaging servers to provide messaging services to a group of client devices is shown.

[0028] Figure 3A A block diagram is shown of an architecture for providing an extension application for use with an instant messaging application according to one embodiment.

[0029] Figure 3B An example of a user interface of a message bubble created at least in part by an extension application is shown according to one embodiment described herein.

[0030] Figure 3C An example showing portions of a template-based message bubble.

[0031] Figure 4A A flowchart illustrating a method for browsing, selecting, and launching an extension application from within an instant messaging application according to one or more embodiments described herein, according to one embodiment.

[0032] Figure 4B An example of a user interface of an instant messaging application is shown, including a browsable view of installed extension applications according to one embodiment described herein.

[0033] Figure 5A An example of a user interface of an instant messaging application is shown, including a view of an extension application within the user interface of the instant messaging application.

[0034] Figure 5B The user interface of the extension app is shown after a user has created content in the extension app, where a view of the extension app is hosted within the user interface of the instant messaging app.

[0035] Figure 5C An example of a user interface of an instant messaging application on a communication device is shown after a user has created content using an extension application and sent the content using the instant messaging application for sending the content.

[0036] Figure 5D An example of a user interface of an instant messaging application on a receiving device is shown after the receiving device receives content created by an extension application on a sending device.

[0037] Figure 5EAn example of a user interface on a receiving device is shown when a user of the receiving device selects a message bubble containing content created by an extension application on a sending device so that the user of the receiving device can edit the content on the receiving device.

[0038] Fig. 5F An example of a user interface providing download and installation of an extension application for use with an instant messaging application on a device is shown.

[0039] Figure 6 A flowchart illustrating a method that may be performed on a receiving device according to one embodiment described herein.

[0040] Fig. 7A Flowchart showing a method according to one embodiment in which an extension application can change its view hosted by an instant messaging application according to one embodiment.

[0041] Figure 7B An example of a user interface of an extension application in an extension view is shown according to one embodiment.

[0042] Figure 8 An example is shown in which extension applications on two or more devices can interact with each other through an instant messaging application on each device according to one embodiment.

[0043] Fig.9A A flowchart illustrating a method for allowing interaction between extension applications on different devices according to one or more embodiments described herein.

[0044] Fig. 9B An example of a message bubble in a user interface that can be displayed when two or more extension apps are interacting in a session Fig.9A The method shown causes.

[0045] Fig. 9C An example of a message bubble is shown, according to one embodiment, where download and installation for a desired extension application occurs while the instant messaging application remains in the foreground.

[0046] Fig.9D , Fig.9E , Fig.9F , Figure 9G and Figure 9H An example of message logging on two different devices participating in a conversation between two instant messaging applications (one on each device) involving a conversation between two extension applications is shown. Fig.9D , Fig.9E , Fig.9F, Figure 9G and Figure 9H The user interface shown illustrates how content in a message bubble created by an extension app can be converted to breadcrumbs according to one or more embodiments described herein.

[0047] Fig.10 is a flow chart illustrating a method according to one embodiment described herein.

[0048] Fig.11 is a flow chart illustrating a method according to one embodiment described herein.

[0049] Fig.12 A user interface of an instant messaging application is shown in which content from two or more extension applications is present within the same message record provided by the instant messaging application.

[0050] Fig.13A A flow chart illustrating a method for providing backward compatibility for legacy devices or legacy messaging systems according to one embodiment.

[0051] Fig. 13B An example of a user interface that has provided backward compatibility within an instant messaging application is shown.

[0052] Fig.14A A method according to one embodiment is shown in which a receiving device (which has received a message) downloads and installs a required extension application to inspect or interact with the content of one of the messages.

[0053] Fig. 14B An example of a user interface showing an extension app marketplace or service from which extension apps can be downloaded and installed.

[0054] Fig.15 An example of a messaging system for providing an application extension using stickers on both a sending device and a receiving device is shown according to one embodiment.

[0055] Fig.16A To illustrate that according to one embodiment, Fig.15 Flowchart of a method performed in the system shown.

[0056] Fig. 16B To illustrate that according to one embodiment, Fig.15 A flow chart of another method performed in the system shown.

[0057] Fig. 16CA user interface according to one embodiment of an instant messaging application hosting a view of a sticker extension application that has been used to create stickers and apply stickers to messages in the instant messaging application's message log is shown.

[0058] Fig.17 is a block diagram illustrating an exemplary API architecture that may be used in one or more of the embodiments described herein.

[0059] Fig.18 A block diagram illustrating a software stack that may include one or more applications and services in an operating system is shown.

[0060] Fig.19 An example of a data processing system that may be used in one or more embodiments described herein is shown. DETAILED DESCRIPTION

[0061] Various embodiments and aspects will be described with reference to the details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding of the various embodiments. However, in some instances, well-known or conventional details are not described in order to provide a concise discussion of the embodiments.

[0062] "One embodiment" or "embodiment" mentioned in this specification means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment. The phrase "in one embodiment" appearing in different positions in this specification does not necessarily refer to the same embodiment. The process shown in the figure below is performed by a processing logic component, which includes hardware (e.g., circuits, special logic components, etc.), software, or a combination of the two. Although the process is described below in accordance with certain sequential operations, it should be understood that some of the operations described can be performed in different orders. In addition, some operations can also be performed in parallel rather than in sequence.

[0063] Various embodiments described herein relate to a messaging system, such as a text messaging system, or a "chat" messaging system, or other systems that allow devices to transmit messages between devices. For example, iMessage by Apple Inc. (Cupertino, California) is an example of a messaging service for iOS devices and Mac (OS X) computers. Typically, a messaging system includes a plurality of client devices, each of which includes at least one instant messaging application and a set of one or more messaging servers that can receive messages from the client devices and transmit messages to the client devices. Figure 1AAn example of a user interface for an instant messaging application on a client device is shown. The client device may be a communication device 10, which may be a smartphone, or a tablet computer, or a desktop computer, or a laptop computer, a wearable onboard computer, or other data processing system or other consumer electronic device. In one embodiment, the device may include a conventional touch screen that can display images and also receive touch input from a user. The touch screen 12 on the communication device may display a user interface for the instant messaging application, which may include a message log 16 and an on-screen keyboard 20 below the message log 16. In addition, in one embodiment, the user interface for the instant messaging application may include a username 14 indicating the recipient of the message sent from the communication device 10. In addition, the user interface may include a text input area 18 indicating the content of the text entered by the user before it is sent; in a sense, the text input area 18 is a text staging area that indicates that the text is ready for sending to the recipient.

[0064] Figure 1B A more detailed example of a user interface for an instant messaging application on a communications device 10A is shown. In this embodiment, the user interface is displayed in a touch screen 12A and includes an on-screen keyboard 20A, a text entry area 18A, a message log 16A, and a user name 14A showing the names of other users to whom messages are sent and to whom messages are received. The text entry area 18A is a temporary storage area for one or more of content, such as text, stickers, extended application content, images, etc., which is ready to be sent in response to a user selection of a send command (and in one embodiment, the content can be edited in the temporary storage area). Figure 1BIn the example shown, a message from Freddy (user name 14A) is displayed on the left side of the message record 16A, and a message sent by the user of the communication device 10A is displayed on the right side of the message record 16A. Thus, the message bubble 17 shows the message "Where are you?" sent by the user of the communication device 10A to Freddy as a reply to the message from Freddy "How about dinner tonight?" The message in the message bubble 17 causes Freddy to reply with "How about?" as shown in the message bubble 19. The use of the word "bubble" such as message bubble or conversation bubble is not intended to imply any specific shape or form; rather, it is intended to represent a dividing line of any shape or form between messages between two or more participants, and thus the dividing line may use a square or line or a message container or a different color, etc. Therefore, the phrase "message bubble" is intended to cover all such dividing lines (or other distinguishing means) between messages between two or more participants, particularly in the context of such dividing lines or other distinguishing means in a message record. In one embodiment, the message log is scrollable, and the messages in the message log are presented sequentially according to their time, so the user can view the actual chat or conversation over time by scrolling the view up or down. Figure 1B The user interface shown in FIG. 1 also includes three icons 22, 23, and 24 located above the screen keyboard 20A and to the left of the text input area 18A. According to one or more embodiments described herein, the extended application view icon 22, after being selected, causes a view of installed extended applications that can operate with the instant messaging application to be displayed, and the view provided can be a browsable view, such as Figure 4B , to allow the user to scroll through multiple pages of all installed extension applications that are configured to operate with the instant messaging application. In one embodiment, the imaging application icon 23 can be an icon that, upon selection, causes the launch of a plug-in for the instant messaging application that provides image creation within the instant messaging application process, such as Figure 5A , Figure 5B and Figure 5C In one embodiment, the camera application icon 24, when selected, may cause the communication device 10A to enter a camera mode in which the device's camera may capture still or video images that may be placed in a message in order to send the image or video.

[0065] Now combine Figure 25 provides a brief overview of an example of a messaging system. Messaging system 50 may include multiple client devices, such as client devices 53 and 54. According to one or more embodiments described herein, each of these client devices may include at least one instant messaging application configured to operate with an extension application, and to transmit at least a text message and optionally a resource locator, or an image, or other content (e.g., relative to a device that is not compatible with the extension application architecture in the instant messaging application). Fig.13AIn a typical messaging system, there may be millions of client devices communicating through a group of message servers. In one embodiment, a plurality of message servers may be configured to receive encrypted messages from a sending device, and then transmit those encrypted messages to a receiving device. Another group of servers may be configured to receive non-text content, such as images or other "attachments", and in a download operation, respond to requests from those receiving devices for obtaining images or attachments and provide those images or attachments to the receiving device. In one embodiment, the sender's output message is encrypted separately for each of the recipient devices in the recipient device. In one embodiment, an asymmetric RSA encryption algorithm may be used to perform encryption. In one embodiment, the public RSA encryption key of each of the receiving devices in the receiving device may be retrieved from a directory service (maintained by one or more message servers), which includes a database, such as a database 52 coupled to the one or more message servers 51. When a client device such as a client device 53 attempts to send a message to another client device, it identifies the other client devices (such as by an email address, or a telephone number, or other identifiers) as one or more message servers 51. The identifier is sent from a client device, such as client device 53, to one or more message servers 51, which then perform a lookup operation in a database 52 based on the provided identifier to retrieve a public key corresponding to the identifier. The public key is then transmitted back to the client device requesting the public key for the particular receiving device, and the client device can then encrypt the message using the public key or using another key (e.g., a symmetric key) that can be randomly generated and encrypted using a public RSA encryption key for the particular receiving device. In one embodiment, the randomly generated key can be randomly generated on a per-message basis. In one embodiment, the resulting message (one message for each receiving device) includes the encrypted message text, the encrypted message key, and the sender's digital signature, and the resulting message for each receiving device is then uploaded to one or more message servers 51 for delivery to a recipient client device, such as client device 54. In one embodiment, the messaging system 50 can be configured to operate over a "public" network, which includes public WiFi access points (such as WiFi access points in coffee shops, airports, etc.), and the Internet. The instant messaging application on each client device 53 and 54 may also be configured to operate with a "private" network provided by wireless cellular telephone carriers such as Verizon and AT&T, and the instant messaging application may be configured to seamlessly switch between use of the private network and the public network depending on the availability of each network and depending on the compatibility of each of the client devices in the messaging session.In one embodiment, the message server 51 may include a set of push notification servers that receive uploaded text messages and "push" those text messages to the receiving device.

[0066] In one embodiment, the messaging system on the client device includes an instant messaging application and one or more extension applications that each operate as an independent process. In one embodiment, the messaging application and the one or more extension applications can each be an independent sandbox process that operates or executes in its own storage space. In addition, the instant messaging application can also communicate with plug-ins such as Figure 5A The image creation plug-in shown operates together, and the plug-in operates in the same process and storage space as the instant messaging application. The instant messaging application and each extension application communicate with each other through inter-process communication such as the XPC framework provided in iOS and Mac OS X. The instant messaging application is designed to receive text about the sending device from the user of the device, and display the text in the message record, and send the text to the receiving device through a set of one or more message servers, and the receiving device displays the received text in the message record on the receiving device through the corresponding instant messaging application on the receiving device. The receiving device and the sending device can each have a copy of the same extension application, which is configured to create a certain type of content according to a specific extension application (or, in an alternative embodiment, each can include a copy of a different extension application compatible with the content they use).

[0067] Figure 3A An example of a software architecture in which an instant messaging application and one or more extension applications operate together to provide an enhanced messaging system is shown. Figure 3A As shown, the messaging system 75 includes an instant messaging application 76 and a set of plug-in modules, such as a combination module 77 and a data transmission module 79 configured to communicate with one or more extension applications 83 via inter-process communication (IPC) 81. Figure 3AAs shown, the instant messaging application and the combination module 77 and the data transfer module 79 operate within the instant messaging application process in a memory space that is controlled by a kernel on a communication device executing the messaging system 75. When a message bubble displays or otherwise presents content transmitted to the instant messaging application via IPC 81, the combination module 77 includes the content of the message bubble. The data transfer module 79 transmits the content and other data to the extended application via IPC 81, and receives the content and other data from the extended application via IPC 81. In one embodiment, both modules 77 and 79 may have an extensible plug-in architecture that allows new plug-ins for new extended applications to be added that generate new content or require new data transfer processes. In this context, a plug-in is additional software that operates within the same process as the instant messaging application. The combination module 77 may use templates to construct message bubbles, such as those described below with respect to Figure 3C The "MSMessageTemplateLayout" described above. The kernel may include a software library or software framework for providing IPC 81 to allow communication between the message system 75 and one or more extension applications 83. In one embodiment, the IPC framework may include a system area called an extension point, which provides an API to allow communication between two different processes and enhance policies in terms of the types of communication allowed. In one embodiment, communication via IPC involves placing (writing) content in a storage area through one process, and the IPC framework allows another process to read from the storage area. In one embodiment, the instant messaging application 76 can automatically start the extension application and can manage its life, including the end of those processes. In one embodiment, each extension application in the extension application 83 runs in its own address space communication between the extension application and the instant messaging application using IPC mediated by the system framework, and they cannot access each other's files or storage space. In one embodiment, each extension application can be a sandbox process separated from each other, and the instant messaging application 76 can also be an independent sandbox process separated from the sandbox process of the extension application. In addition, the extension application can be set with fewer system permissions than the instant messaging application, so that the extension application operates in a more restricted environment than the instant messaging application. More information about inter-process communication using extension reference programs can be found in U.S. Patent Application No. 14 / 488,122 filed on September 16, 2014 and published as U.S. Patent Publication US2015 / 0347748, which is incorporated herein by reference.

[0068] In one embodiment, the messaging app provides a view of content obtained from the extension app through inter-process communication. The extension app can create the content in its own process and then provide the content in a format that the messaging app is known to receive (such as a standard image format or other standard format). This allows the messaging app to then present the content from the extension app in one or more message bubbles in the message record (without having to execute the extension app on at least the receiving device). Figure 3B An example of a message bubble 17A containing content created and provided by an extension application is shown, as shown by content 85, which may also include a text message created or provided by the extension application, such as text message 86. In one embodiment, the message bubble 17A may also include an icon 87, which may be an icon of the extension application that created the content 85.

[0069] In one embodiment, the object created by the extension application is shown in the message record on the sending device and the receiving device without launching the extension application. The extension application should provide enough information to build a message bubble as part of the object. The object may include some opaque data encoded in the resource locator and a layout specification provided as an MSMessageTemplateLayout object. MSMessageTemplateLayout is a subclass of MSMessageLayout and represents a method of specifying the layout of a message bubble.

[0070] In one embodiment, MSMessageTemplateLayout may have the following properties, such as Figure 3C As shown:

[0071] 1) Image or Media File URL: The image is provided as a UIImage or file URL to an image file or as a file URL to a video

[0072] 2) Image title: a string that will be displayed on top of the image or movie

[0073] 3) Image Subtitle: A string that will be reproduced on top of the image or movie below the image title

[0074] 4) Title: The string that will be reproduced in the title bar below the image or movie

[0075] 5) End Title: The string that will be reproduced on the right side aligned with the title bar below the image or movie

[0076] 6) Subtitle: The string that will be reproduced in the title bar below the title

[0077] 7) End subtitle: The string that will be displayed on the right side aligned with the title bar below the end title

[0078] 8) Extension application icon: This is not part of the MSMessageTemplateLayout, but is derived from the bundle identifier of the extension application created by MSMessage.

[0079] Instant messaging applications can use this information to construct messages similar to Figure 3C Example message bubbles shown.

[0080] The MSMessageTemplateLayout is serialized and transmitted to the remote device along with the opaque data. When received on the receiving device, the instant messaging application creates an MSMessageTemplateLayout using the serialized data and uses the template layout to draw the message bubble in the recipient's message history.

[0081] In one embodiment, an extension app configured to operate with an instant messaging app is not executable outside of the instant messaging app, and therefore its lifecycle is completely managed by the instant messaging app. In addition, as further described below, in one embodiment, the download and installation of the extension app can be controlled solely by the instant messaging app.

[0082] In one embodiment, each extension app is available from an application marketplace or distributor such as the Apple App Store (trademark) for messaging extension apps, and can be launched from within the instant messaging application. Figure 4A An example of a method according to one embodiment is shown, in which installed extension applications can be browsed from within an instant messaging application, and a specific extension application can be launched to allow a user to interact with the specific extension application. The method can begin with operation 101, in which the instant messaging application displays its message history and an on-screen keyboard. Figure 1B An example of a user interface of such an instant messaging application is shown. Then in operation 103, the instant messaging application can receive input to display a browsable view of the installed extension application within the view of the instant messaging application. For example, the user can tap icon 22 ( Figure 1B ) to select the icon, which in turn causes the browsable view to be displayed in operation 105. In one embodiment, the browsable view of installed extension applications selects an alternative on-screen keyboard, and retrieves a list of installed extension applications from an execution registry, causing an icon of each installed extension application in the installed extension applications to be displayed. Figure 4B An example of the result of operation 105 showing the browsable view 157 and replacing the on-screen keyboard of the instant messaging application is shown as follows Figure 4B shown.

[0083] See also Figure 4B , it can be seen that the browsable view 157 includes a plurality of icons each representing one of the installed extension applications operable with the instant messaging application, the instant messaging application providing such Figure 4B The user interface shown. Figure 4B The user interface of the instant messaging application shown includes a message record 153 shown in the upper portion of the touch screen 151 on the communication device 150 executing the instant messaging application. The user name 155 of the other user (the recipient of the message sent from the communication device 150) is shown at the top of the user interface of the instant messaging application. The text input area 155 (which displays text or other content that has been temporarily stored or ready to be sent) is displayed between the message record 153 and the browsable view 157. In one embodiment, the browsable view can be browsed by swiping the user's finger on the touch screen to display pages in various views of installed extension applications (and in one embodiment, it can also show, for example, uninstalled extension applications that need to complete the download process or need to complete the installation process). In one embodiment, the page indicator 159 at the bottom of the user interface can indicate the current page of the installed extension application. In one embodiment, an icon in the icon can be retained to launch or enter the extension application market, an example of which is shown in FIG. Fig. 14B In another embodiment, the extended application marketplace can be displayed in the browsable view 157 in response to selecting icon 167. Figure 4B In the illustrated embodiment, icon 167 is selectable to cause an extended application marketplace such as Fig. 14B The Extended App Marketplace icon is shown. Figure 4BIn the illustrated embodiment, icon 169 may be selected (e.g., the user taps icon 169) to cause a browsable view of recently sent stickers or handwritten messages or other recently sent items or recently used applications, etc. to be displayed. In one embodiment, displaying recently sent items, etc. may be provided by a plug-in operating within the process of the instant messaging application. Other extension applications include a restaurant reservation application icon 161, which may be an extension application that can be used to engage a restaurant reservation service such as Open Table. Another example of an extension application is represented by icon 163, which, when selected, launches a restaurant review application that provides restaurant reviews and is similar to the reviews provided by, for example, Yelp. Other extension application icons 165 and 171 represent other extension applications that have been installed and can be launched by selecting one of those extension application icons.

[0084] See again Figure 4A Once the browsable view is displayed through operation 105, the user can select one of the extension applications by selecting the corresponding icon, which in turn causes the selected extension application to be launched in operation 107. In one embodiment, the instant messaging application calls the system service to launch the selected extension application and uses, for example, Figure 3A The illustrated architecture prepares it to execute as an extension application of an instant messaging application. Once the selected extension application has been launched and executed, an instant messaging application such as instant messaging application 76 can host a view of the content provided by the executing extension application through the IPC framework described herein. For example, in Figure 4A In operation 109 shown in FIG. 1 , the instant messaging application may display a view of the content provided by the extension application within a portion of the view of the instant messaging application. FIG. 5A to FIG. 5F An example of how an instant messaging application can host a view that executes the content of an extension application will now be described.

[0085] Figure 5A An example of an instant messaging application is shown hosting a view of a plug-in for the instant messaging application, the plug-in being an image creation application that is launched by selecting icon 207 (e.g., a user touching or tapping or otherwise selecting icon 207). In another embodiment, Figure 5A The plugins shown in the bottom portion of the user interface shown can be enabled by selecting Figure 4B In the navigation view 157, one of the icons can be started. Figure 5A The example shown may be implemented as a plug-in for an instant messaging application, but in another embodiment, Figure 5A The example shown can be an extended application. Figure 5A In the example shown, the view of the plug-in (or extension application) selects to replace the on-screen keyboard of the instant messaging application, but in one embodiment, the message record of the instant messaging application is still visible and displayed in the user interface, allowing the user to scroll the message record to view the entire record. In another embodiment, the view of the plug-in or extension application is an overlay on the top of the on-screen keyboard, a portion of which may be visible. In one embodiment, the record is displayed on the touch screen 202 of the communication device 200. The user interface of the message application also includes a user name 203 that presents the names of other users in the conversation or chat in one embodiment. The user interface of the message application also includes a text input area 211 similar to the text input area 18A and the text input area 155. The plug-in (or extension application) includes a canvas 215 and various controls and options that the user can select to draw or create an image. In one embodiment, if the sketch option 217 is selected, the drawing control 212 allows the user to select different colors to sketch on the canvas. If the tap option 221 is selected, the plug-in (or extension application) can also provide a tap as a message. If the heartbeat option 219 is selected, the plugin (or extension app) may also provide a heartbeat. In one embodiment, the sketch, heartbeat, and tap are similar to the sketch, heartbeat, and tap in the Digital Touch app available on the Apple Watch. The plugin (or extension app) also includes an expanded view icon 223, which when selected will cause the plugin (or extension app) to open from the Figure 5A Its current compact view is switched to an expanded view such as Figure 7B Expanded view shown. Figure 5A In the example shown, the user has just launched the plug-in image creation application (or in another embodiment, the extension application) and has not yet created any content. Figure 5B In contrast to the state of the plugin (or extension application) shown in Figure 5B In the illustrated plug-in (or extension application), the user has created a sketch of a smiley face using the sketch option 217 or by drawing with the user's finger on the canvas 215. The user can then send the drawing to another user (or multiple users) who are recipients of the message. Thus, for example, the user of device 200 can select a send command from within an instant messaging application executed on communication device 200 to cause the content created by the plug-in (or extension application) to be sent to the recipient. The result of the send operation is shown in FIG. Figure 5C203. Message bubble 230 shows a smiley face thumbnail created by the plugin (or extension application); in the case of an extension application, the created content is sent from the extension application to the instant messaging application via the IPC framework and then presented within message bubble 230 to indicate that a message containing the content is transmitted to one or more recipients. In one embodiment, a delivery indicator 231 may indicate that the message has been delivered and a hold control 232 may be provided to the user to allow the user to retain the content in the message history in those embodiments where the content is automatically cleared after a period of time.

[0086] Figure 5D , Figure 5E and Fig. 5F In response to receiving Figure 5C The content from the communication device 200 shown here occurs on the recipient's device. Figure 6 The flowchart shown may illustrate a method performed on a receiving device such as Figure 5D , Figure 5E and Fig. 5F The communication device 250 is shown. Figure 5D , it can be seen that communication device 250 (used by Freddy) has received the smiley content shown in message bubble 253. The content was created by the extension application executing on communication device 200, which provided the smiley content to the instant messaging application executing on communication device 200, which in turn sent the smiley content to the instant messaging application executing on communication device 200 through a messaging service (e.g., a set of messaging servers such as Figure 2 The content is transmitted to the instant messaging application executing on the communication device 250 by the messaging server 51 shown in the figure, which then presents the content in the message bubble 253. In one embodiment, when the content uses a standard format (including standard image, audio and video formats in one embodiment), the instant messaging application can render the content, and thus the extension application does not need to be installed or executed in order to display the content created by the corresponding (remote) extension application on the sending device. Figure 5D In the illustrated case, the message bubble 253 can present content even if the corresponding extension application on the communication device 250 is not launched or even installed. Figure 5DThe instant messaging application shown includes a message log 201A, a text input area 211, and an on-screen keyboard 255 in its user interface. In one embodiment, content received from a remote extension application will not cause the corresponding extension application on the receiving device to automatically launch, even if the corresponding extension application is installed. In this embodiment, the corresponding extension application on the receiving device can be launched by a user selection of a message bubble containing content created by the remote extension application. If the user does not select the content by, for example, touching or otherwise selecting the message bubble 253, then the extension application corresponding to the remote extension application is launched if it is installed on the communication device 250. Figure 5E In the result shown, the user interface of the extension application has occupied the space where the on-screen keyboard 255 was previously displayed and a smiley face drawing is shown within the canvas 215, thereby allowing the user of the communication device 250 to change or otherwise modify the sketch and send it back to other users in the chat session or conversation. In other words, if the corresponding extension application is not installed on the communication device, then in one embodiment, the instant messaging application may issue a notification to the user asking or offering to install the application for the specific message bubble that has been selected. An example of this situation is as follows Fig. 5F As shown, wherein notification 259 includes two user-selectable options, one of which will install the desired application of the message bubble. In another embodiment, a worksheet showing an information page from an extended application market can be displayed in the view of an instant messaging application.

[0087] In one embodiment, the message transmitted from the remote device to the communication device 250 includes metadata for specifying the remote extension application used to create the content. In one embodiment, the metadata can be an application identifier, such as an identifier provided by an application market or an extension application market from which the extension application can be downloaded and installed, or can be a different identifier associated with the identifier used by the application market. In one embodiment, the notification 259 can be obtained by selecting the message bubble 253, and in another embodiment, if the application identifier in the metadata of the content is not installed when the communication device 250 receives the content of the message bubble 253, then the notification 259 can be automatically obtained.

[0088] Figure 6 Now combine Figure 5D , Figure 5E and Fig. 5FReference is made to explain a method in one embodiment in which a receiving device processes content created by a remote extension application, such as a remote extension application executed on the communication device 200 in conjunction with an instant messaging application. In operation 301, the communication device may receive a message having content created by an extension application, such as a remote extension application on a sending device operating in conjunction with an instant messaging application. In addition, the communication device may also receive metadata, which may include a message bubble identifier, a session identifier, and an extension application identifier, as well as possible other data, such as an optional resource locator and other data (state information) that may be associated with the optional resource locator, and the resource locator may also include state information encoded into the resource locator. More information with respect to the use of resource locators and data associated with resource locators will be described in conjunction with Figure 8 , Fig.9A and Fig. 9B To provide. Then in operation 303, the communication device that has received the message in operation 301 processes the content and displays the content in a message bubble, which is identified by a message bubble identifier, and the message bubble is displayed in the message record. In one embodiment, the processing of the content may include decrypting the content and reproducing the content to be presented and displayed in the message bubble. In one embodiment, the content is displayed by the instant messaging application without the help of the extension application; in other words, the extension application may not be installed on the communication device or may not be executed when installed, and therefore an on-screen keyboard is displayed in one embodiment. Then in operation 305, the communication device receives a selection of a message bubble that displays content created by a remote extension application. In one embodiment, referring to FIG. 5D, the user may tap the message bubble on the touch screen or otherwise select (e.g., the user uses a stylus on the touch screen or uses a mouse with a desktop computer, etc.) the message bubble 253 so that the selection is made in operation 305. In response to operation 305, the communication device, such as the communication device 250, determines whether the extension application has been installed in operation 307. In one embodiment, this can be performed by checking a list of installed extension applications or registration information maintained by the instant messaging application. In one embodiment, the metadata received in operation 301 includes an application identifier, and in operation 307, the instant messaging application searches the list to determine whether the identifier exists in the list. If the identifier does not exist in the list, the instant messaging application determines that the extension application is not installed, causing the communication device to perform operation 309, in which a user may be displayed such as Fig. 5FThe notification 259 shown provides for downloading and installing the application specified by the application identifier, which was received as part of the metadata in operation 301. If the user selects the option "yes", operation 311 is performed, in which the instant messaging application causes the communication device to access an application market (such as a Fig. 14B 9C shows an example of a message bubble 471 that remains in the message history during the download and installation process, wherein the message bubble includes a progress bar 473 indicating the progress of the download and installation operation, while the instant messaging application remains the foreground application displaying the message bubble 471 in the message history of the instant messaging application. In another embodiment of operation 311, a worksheet showing an information page from the extension application market may be displayed above the instant messaging application (optionally still displaying a portion of the instant messaging application), and the worksheet may show a "Buy" or install or download button, which upon selection of the corresponding button may result in the download and installation of the extension application, and the worksheet may be canceled (removed from the display screen) by selecting a cancel command or by selecting Buy or Install or Download. After downloading and installing the extension application, you may continue to Figure 6 Operation 313 shown in , in which the extension application is launched and the content and other data used or created by the remote extension application are transmitted (via IPC) to the extension application through the instant messaging application, and in one embodiment, the extension application is displayed in a compact view or an expanded view, and the content created by the remote extension application is displayed in the view. Figure 6 As shown, if operation 307 determines that the extended application has been installed, operation 313 will be further performed after operation 307 . Figure 5E An example of the result of operation 313 is shown.

[0089] In one embodiment, the instant messaging application can launch different types of extension applications in different ways according to the type of the extension application. For example, in response to receiving a message bubble containing content from an extension application of a specific predetermined type, an extension application of one type can be automatically launched. In one embodiment, other extension applications of different types can be launched only in response to the selection of a message bubble containing content from the extension application or the selection of an icon representing an extension application in a browsable view such as browsable view 571. It may be desirable to allow a specific extension application of a specific type to be automatically launched in response to receiving content displayed in a message record, while not automatically launching other types of extension applications. In another alternative embodiment, one or more extension applications may be allowed to execute in the background and their corresponding user interfaces presented in their corresponding message bubbles may be allowed to be updated, and when the user selects one of the message bubbles, the extension application appears in the foreground (e.g., displaying its UI to replace the on-screen keyboard).

[0090] In alternative embodiments, the metadata may include a format or extension identifier, such as an identifier of an image format, that may be used to determine an available extension application that can handle that image format on the receiving device.

[0091] Fig. 7A and Figure 7B Another aspect of the embodiments described herein is shown, wherein an extension application may cause a change in view by sending a communication to an instant messaging application. In one embodiment, an application programming interface (API) may be provided between the extension application and the instant messaging application to allow the extension application to call the API to change its view within the instant messaging application that hosts the view of the extension application. In one embodiment, the extension application may have at least two different views that may include a compact view and an extended view. In one embodiment, when the message record remains displayed in the user interface of the instant messaging application, the compact view may be a view that selects an alternative to the on-screen keyboard of the instant messaging application. In the extended view, the message record is no longer displayed, and the on-screen keyboard is not displayed, but specific other components of the user interface of the instant messaging application are displayed, such as the text input area 211 and the camera activation icon 235. Figure 7B An example of an expanded view is shown in which canvas 215A occupies most of the space of the touch screen. Figure 7B As shown, the user can select the compact view icon 223A to cause the system to Figure 7B The expanded view shown changes back to a compact view, such as Figure 5AView shown.

[0092] Fig. 7A The method shown in is one embodiment of how the illustration may vary, and it should be understood that in alternative embodiments, the sequence of operations may be performed in a different order, and there may be omitted steps or intervening or additional steps.

[0093] exist Fig. 7A In operation 351, the extension application may be displayed by an instant messaging application having a specific view or style. In operation 353, the extension application may call the instant messaging application to obtain the current presentation view / style of the extension application. In operation 357, the instant messaging application may provide a return result for the call from operation 353, and the return result may indicate the current presentation view / style of the extension application. In response to the current display received in operation 357, the extension application may request a change in the presentation view / style by providing the instant messaging application with a call to cause the change to occur, and receive the call in operation 361. In one embodiment, the instant messaging application may initially respond to the call with a confirmation that the view is changing or will change. In operation 363, in response to the call, the instant messaging application changes the presentation style / view and displays the extension application within the requested view presentation view / style, and in operation 365, the instant messaging application notifies the extension application that the change in view is complete. See again Figure 7B If the user selects the compact view icon 223A, a call will be made from the extension application to the instant messaging application to change the view of the extension application within the user interface of the instant messaging application.

[0094] The present invention will now be described with respect to another aspect of the embodiments described herein. Figure 8 , Fig.9A , Figure 9B and Fig. 9C In one aspect of this embodiment, an extension application and a corresponding extension application on another device can participate in a communication session and exchange information back and forth in their communication session, and all of these operations occur in the context of message records maintained by two instant messaging applications that interact between the two extension applications, such as Figure 8 In one embodiment, the plug-in of the instant messaging application can also operate in a similar manner, and information is exchanged between the plug-ins through the two instant messaging applications in the session. Figure 8The message system 400 shown includes at least two client devices, namely, a client device 401 and a client device 405, and also includes a group of one or more message servers 403. The client devices 401 and 405 are similar to Figure 2 The client devices 53 and 54 in the embodiment, and a set of one or more message servers 403 may be similar to Figure 2 A set of messaging servers 51 in the example embodiment of the present invention. Each client device may include an installed copy of a particular extension application (such as, for example, an extension application for making reservations at a restaurant), and the extension application on each device may be used to create content (e.g., text, images, audio, video, etc.), and the content is transmitted to the instant messaging application on the device through the inter-process communication framework for a particular message, which may be referred to as a message bubble in the message history on the particular client device. The messaging application receives the content (and optionally receives other data from the extension application, including, for example, an identifier of the extension application, a resource locator, and metadata optionally used by corresponding or remote extension applications on other devices, etc.) and displays the displayable content (such as an image of a restaurant provided by the extension application in a message bubble in the message history, the extension application being a restaurant reservation application such as "Open Table"). In effect, the instant messaging application hosts a view within the instant messaging application, and the content of the view is provided by the extension application. In one embodiment, the resource locator and metadata are opaque to the instant messaging application (e.g., cannot be identified by the instant messaging application), but can be used by the extension application on each device to maintain state information of the session between the extension applications, and by transmitting the resource locator and metadata between the extension applications, the instant messaging application on each device serves as a communication mechanism between the extension applications. In one embodiment, the state information about the session can be encoded in the resource locator, or can be provided in the metadata, or both. In one embodiment, the content created from the session on each device is displayed in the same single message bubble in the message record (identified by a session identifier that can be maintained by the instant messaging application), and each time the content changes (based on a change of device), the updated content continues to be displayed in a single message bubble in the message record, and any previous message bubbles displaying content in the session can be converted to breadcrumbs, and these previous message bubbles will also include the same session identifier as the latest message bubble. Now refer to both Fig.9A To describe Figure 8 The operation and functions of the components shown, Fig.9A A method of operating a messaging system 400 in one embodiment is shown.

[0095] exist Fig.9A In operation 451 of , an extension application such as extension application 407 may create content and generate a resource locator and data (or may modify existing content, resource locators, or data). The extension application may be similar to extension application 83 and in one embodiment executes in one process while an instant messaging application, which may be similar to instant messaging application 76, executes in another process and communication between processes occurs through an IPC (such as IPC 81), which may be a software framework or library for providing inter-process communication between two different processes. Extension application 407 may be, for example, a restaurant reservation application that creates a reservation through a website, which may provide status information for use with the resource locator (or the status information may be encoded into the resource locator). In one embodiment, the data with the resource locator may be status information provided from the website, and the status information may include information about a particular restaurant and a reservation time and number of reservations. Extension application 407 may present a user interface for making a restaurant reservation through a website in a compact view or an expanded view while displaying the rest of the instant messaging application, including, for example, a message log. Thus, the user of extension application 407 can see the context of the conversation in the message log while interacting with the extension application and the website (through the extension application) to create a restaurant reservation. In one embodiment, the user can browse various restaurants (within the restaurant reservation application) and search for restaurants. After the user of extension application 407 has selected a restaurant and entered a reservation, extension application 407 can transmit the content along with the resource locator and data 417 shown in FIG. 8 to the instant messaging application on device 401 via IPC 415. This process is as follows Fig.9A407 and the session identifier. In operation 454, the instant messaging application 409 associates the content received from the extension application 407 with the message bubble and displays the content in the bubble of the message record in the user interface of the instant messaging application 409. Then in operation 457, in response to the send command received from the user, the instant messaging application 409 sends the message (if any) and the content received from the extension application 407, as well as the application identifier identifying the application 407 (and optionally the icon of the application 407) and the resource locator and data (if any) and the session identifier to the second device through one or more messaging servers 403, and the one or more messaging servers 403 transmit the communication 419 to the second device 405 (also referred to as the client device 405). In one embodiment, operations 453 and 455 may occur as part of operation 457 in response to the instant messaging application receiving a selection of the send command. The icon of the application 407 may be displayed on the message bubble of the receiving device even when the corresponding extension application is not installed; see, for example, Figure 3B Icon 87 in. Fig.9A In operation 459 , as shown, the instant messaging application 411 on the client device 405 receives content from the one or more messaging servers 403 and displays the content in the identified message bubble 421 within a message record provided by a user interface of the instant messaging application 411 . Fig. 9B A more detailed example of such a message bubble 471 is shown in one embodiment with content created by a restaurant reservation extension application. Fig. 9BIn the example shown, the content includes the name of the restaurant, the reservation time, and the number of people in the reservation. In one embodiment, the content can be displayed by the instant messaging application without launching the extension application 413. In one embodiment, the extension application 413 is not launched until the user of the client device 405 selects the message bubble 421, thereby indicating to the client device that the user of the client device 405 intends to interact with the content in the message bubble 421. In an alternative embodiment, the extension application 413 can be launched when the content is received by the instant messaging application 411, but remains in the background and is ready to execute when the user of the client device 405 enters a command to cause the extension application to appear. In operation 461, the extension application 413 identified by the application identifier provided in the communication 419 is launched, and if it is not already ready to be launched, in response to the selection of the message bubble 421, the instant messaging application 411 transmits the content associated with the message bubble 421 and the resource locator and data 425 to the instant messaging application 413 via IPC 423. In one embodiment, extension app 413 is a corresponding extension app that is the same extension app as extension app 407 , while in another embodiment, they are merely compatible in that they can handle the same type of content.

[0096] Here, the extension app 413 can receive user input from the user of the client device 405 and can modify one or more of the content, resource locator, or data. For example, the user of the client device 405 can cause the extension app 413 to access one or more websites to make a modified restaurant reservation by modifying the time, number of people, a specific restaurant, etc. In one embodiment, the extension app 413, as well as the extension app 407, can also interact directly (but separately and independently) with the network server by sending the resource locator and data to the network server and receiving a response from the network server that may include modified data or modified resource locators or new data and / or new resource locators, etc. In one embodiment, the network server can store data used during the session, and this stored data can include information for some or all of the state information that can also be maintained by both extension apps in the session. Similarly, if the presented extension app 413 is displayed in a compact view, the user of the device 405 can interact with the extension app 413 to make a restaurant reservation while the context and conversation of the chat or messaging session is displayed in the message log of the instant messaging app 411. The user of the client device 405 can browse the message history while continuing to view and interact with the extension application 413. Thus, the extension application 413 can receive the user input in operation 463 and can modify at least one of the content, resource locator, or data, and can then transmit the resource locator and data 427 (which can be modified or new) to the instant messaging application 411 in operation 465. The instant messaging application 411 can then send the content (which can be modified) along with the application identifier and resource locator (which can be modified) and data (which can be modified) and bubble ID back to the client device 401 in operation 467. As shown in operation 469, this process can be repeated over time as two users can engage in setting up restaurant reservations in the examples provided herein.

[0097] It should be understood that many different types of extension applications can provide a collaborative environment between users of client devices 401 and 405 to exchange information and work together, and that restaurant reservations is one such extension application. Figure 8 and Fig.9AThe restaurant reservation example described is merely an example of such an extension application that can provide a collaborative environment in the context of the user interface of an instant messaging application. Examples of other types of extension applications that can provide a similar collaborative environment include, for example: sticker extension applications; imaging applications; drawing applications; content creation applications; games; music creation applications; content consumption applications; voting applications; map applications, etc.

[0098] In one or more embodiments, Figure 8 shown and relative to Fig.9A The collaborative environment may utilize breadcrumbs, and each breadcrumb in the breadcrumbs may be identified by a session identifier. The breadcrumb represents a converted message bubble and is identified by a session identifier that is shared with other message bubbles in the session. In one embodiment, each previous message bubble identified as a new message bubble by the same session identifier may be converted to a breadcrumb that may be displayed differently from the original content shown in the message transcript. In one embodiment, as each new message bubble in the session arrives or is added to the message transcript, previous message bubbles identified by the same session identifier may be converted to a breadcrumb and shown in the message transcript, such as Fig.9D Figure 9E Fig.9F , Figure 9G and Figure 9H shown. Fig.9D and Figure 9H shows a message log 510 on Joe's device, and Fig.9E , Fig.9F and Figure 9G Message log 17 is shown on Lester's device. Message log 510 shows Lester's username 511, and message log 517 shows Joe's username 518. Fig.9D , Fig.9E , Fig.9F , Figure 9G and Figure 9H In the example shown, Lester and Joe are engaged in a text messaging conversation and each uses an extension application, such as an image creation application or other extension application. For example, Lester can use Figure 8 The extended application 407 is shown, and Joe can use Figure 89D , it can be seen that message transcript 510 includes message bubble 512 and message bubble 513 indicating the content of the conversation within message transcript 510. In addition, Joe has used extension application 413 to create content 515 that is appended to message bubble 514. For example, Joe may have entered text as a text message and also used extension application 413 to create content, and then had both text displayed within message bubble 514, and content 515 (which, in one embodiment, is associated with or is part of message bubble 514) sent to Lester's device. The right side of message transcript 510 shows messages sent by Joe, while the left side of message transcript 510 shows messages received from Lester. Now referring to Fig.9E , it can be seen that message bubble 513 is now on the right side of message record 517, while message bubble 514 and content 515 are on the left side of message record 517 on Lester's device. Therefore, Lester's device has received the text message within message bubble 514, and also received content 515 generated by the extension application on Joe's device. Lester can then tap content 515 to cause the corresponding or corresponding extension application on Lester's device to launch. Content 515 is associated with a session identifier maintained by the extension application on Lester's device. For example, in one embodiment, in this process, as a result of the user selecting content 515, Fig.9A The illustrated operation 461 may be performed on Lester's device and the content 515 may be displayed in a message bubble. Lester may then use the extension application on Lester's device to create modified or new content and send the modified or new content back to Joe. Fig.9F , it can be seen that Lester has created modified content or new content 521 which is displayed in a staging area 519 which shows text and other content that is ready to be sent and will be sent in response to selection of a send command by the user, such as Fig.9F When Lester selects the send command with the content shown in the scratch area 519, this causes the text message 520 and the new or modified content 521 to be sent to Joe and can be viewed from Figure 9G517, which shows a text message 520 in a message bubble 520A to the right of the message transcript 517, and also shows content 521 (in one embodiment, the content is associated with or is part of the message bubble 520A), which is modified or created as new content by Lester using the extension application 407 on Lester's device.

[0099] Available from Figure 9G As can be seen in the figure, content 515 has now been converted to breadcrumbs 515A. In one embodiment, this conversion can be performed by the instant messaging application or alternatively by the extension application. In one embodiment, the extension application can provide text that appears within the breadcrumbs 515A, and the instant messaging application will use the session identifier to identify one or more message bubbles to be converted to breadcrumbs, and in one embodiment, this will cause content 515 to be converted to breadcrumbs 515A and display the breadcrumbs next to the associated message bubble 514 without converting the message bubble 514 text message to breadcrumbs. Therefore, Figure 9G Shows how a session identifier associated with a previous message bubble or content created by an extension application can be used to convert one or more previous message bubbles into breadcrumbs on a sending device. Figure 9H FIG. 4 shows how the conversion is displayed on the receiving device in one embodiment. Figure 9H In the illustrated figure, message bubble 520A is shown on the left side of message transcript 510 along with new or modified content from extension application 403 on Lester's device. Content 515 on Joe's device has been converted to breadcrumbs 515A on the right side of the message transcript and is displayed adjacent to message bubble 514, which accompanied content 515 when it was originally sent.

[0100] If, in operation 459, the receiving device, such as client device 405, is capable of installing and using the extension application (identified by the application identifier provided in communication 419), but the extension application is not installed on the receiving device, the receiving device may offer to download and install the extension application (also specified by the application identifier in communication 419 on the receiving device) within the user interface of the instant messaging application. Fig. 9C An example is shown where content from an extension app 407 can be displayed on a client device 405 in a message bubble 471, and the client device 405 is in the process of downloading and installing the extension app identified by the app identifier in communication 419. Fig. 9CIn the example shown, while the extension application 413 is being installed, the content is displayed within the message bubble 471. In one embodiment, during the installation process, a progress bar 473 (or alternatively a progress ring) may be displayed within the message bubble 471. In one embodiment, the download and installation process may be performed in the background while the instant messaging application remains the foreground application. If the receiving device is unable to install or use the extension application, in one embodiment, the resource locator and metadata may be transmitted to a web browser on the receiving device, and the web browser may become the foreground application and allow the user to interact with the web page referred to by the resource locator.

[0101] In some embodiments, it is desirable to provide an identifier of each user to each extension application executing on a client device, particularly in the case of a collaborative environment where two or more users interact through an instant messaging application and extension applications. Fig.10 An example of a method for providing an identifier for each extension application without affecting the loss of user privacy is shown. Fig.10 The method shown may be performed by each instant messaging application on each client device. In one embodiment, the method may be performed in response to a call to an application programming interface (API) from an extension application, which allows the extension application to request an identifier of a local user. In one embodiment, the identifier may be an email address of the local user, or a phone number, code, or other identifier used by the messaging system. In one embodiment, the identifier provided to the extension application is a local user's email address. Fig.10The fuzzy identifier created by the method shown in FIG. 501. In operation 501, the instant messaging application may generate a salt value for a specific extension application in response to a call from an extension application. In one embodiment, the salt value may be a random number associated with the specific extension application. Then, in operation 503, the instant messaging application may generate a hash (such as a SHA-1 hash) of a combination of a user or device identifier and a salt value. For example, the user identifier may be a user's phone number or email address, and the phone number or email address may be combined with a salt value, and then a hash of the combination may be created in operation 503. Then in operation 505, the hash is provided to the extension application via IPC, and the hash value may be used as data together with a resource locator, and then the resource locator may be sent to other extension applications to identify the user who has changed or created the content. In another embodiment, the instant messaging application may obfuscate the identifier by maintaining a mapping of the identifier to a randomly generated unique identifier for each extension application. In other words, for a given extension application, the instant messaging application can generate a random (and unique) identifier for the extension application, and associate (e.g., map) the random identifier to an identifier of the user (e.g., a local user's email address, or phone number, or other identifier used by the messaging system). This random identifier for a given extension application can be provided to the extension application, but the identifier is not provided to the extension application. Another extension application will receive a different randomly generated identifier. This particular extension application identifier can then be provided to a corresponding extension application on another device, so that the two extension applications can track the user who performed a specific action in the context of a conversation or other collaborative environment.

[0102] In one embodiment, the messaging system described herein can confirm that one or more extension apps received the message, and this can be useful in certain situations where the extension app needs to ensure that the remote extension app has the same known state as the local extension app. Fig.11An example of a method for providing confirmation of receipt is shown. In operation 551, the instant messaging application receives a "send" command from the user and responds to the local extension application providing content to the instant messaging application. Then, in operation 553, the instant messaging application sends the message and message bubble identifier and the content and resource locator (if any) to a receiving device that also includes a messaging application through one or more messaging servers. At a certain moment, the instant messaging application on the local device receives confirmation of receipt of the message and the content and optional resource locator in operation 555, and then in operation 557, the confirmation of receipt is transmitted to the local extension application through the ICP, so that when the extension application provides its content for transmission to the remote extension application, the local extension application recognizes that the remote extension application has the same known state.

[0103] Fig.12 Another aspect of the embodiments described herein is shown and relates to presenting multiple message bubbles within a message log, wherein different message bubbles have content created by different extension applications, and at least one of the message bubbles can be executed (and have its content displayed in a compact view) while the message log of the user interface of the instant messaging application is also displayed. Fig.12 An example of such an aspect is shown in FIG. 6 . The communication device 600 in this embodiment includes a touch screen 601 displaying a message log 603 including two message bubbles 605 and 607. In addition, the instant messaging application presents a compact view 609 of an extended application, which in this case is an extended application for restaurant reservations. Fig.12 In the example shown, the extension application for restaurant reservations has been used to send a message displaying restaurant reservation content, as depicted by message bubble 607. This may occur as a result of receiving a message from another user using another communication device that provides content for a restaurant review from another extension application. Fig.12 In the example shown, message bubble 605 shows content created by a remote extension application that provides restaurant reviews that are different from an extension application for making restaurant reservations. In one embodiment, both extension applications are executed on a communication device as part of a messaging session or chat.

[0104] Another aspect of the embodiments described herein relates to backward compatibility and is shown in FIG. 13A and Fig. 13BSome older devices may not be compatible with the extension application architecture described herein, or may not use the architecture or may not even be smart phones. In one embodiment, the client device sending the message can automatically or based on a determination that the receiving device is not compatible with the extension application, provide alternative content. A method of providing backward compatibility is as follows Fig.13A In operation 651, the extension application on the sending device creates content on the sending device (first device) for delivery to the second device via the instant messaging application and messaging service. This operation is similar to Fig.9A The content created in operation 651 may then be transmitted to the instant messaging application via inter-process communication in operation 653 in response to the instant messaging application receiving the user's "send" selection. Then, in operation 655, the instant messaging application may display the content on the first device and also determine that the second device is not compatible with the extension application. In one embodiment, this may be determined as a result of receiving information about the receiving device (the second device) from one or more messaging servers, such as Figure 2 The one or more message servers 51 shown may maintain information about the status of each device, such as the version of the device's operating system or the type of device, etc. As a result of determining that the second device is incompatible with the extension application, the instant messaging application may send alternative content to the second device in operation 657, and Fig. 13B An example of this alternative content is provided.

[0105] Fig. 13B The communication device 675 shown may be, for example, an old smartphone using an old operating system that is incompatible with the extended application described herein. However, the communication device 675 includes a touch screen 679 and a text input area 683 and an on-screen keyboard 685 for providing the functionality of an instant messaging application, which also displays a message record 681 including a message bubble 687. The message bubble 687 contains alternative content provided by the sending message application on the remote device. In this case, the content includes an image 689 and a resource locator 691 that can be selected by the user to display the web page pointed to by the resource locator 691. In other words, the resource locator 691 can be selected by the user to invoke a web browser on the communication device 675 to allow the user of the communication device 675 to interact with a web page through the web browser, which web page can have the same role as the extended application in some cases in interacting with a website.

[0106] Another aspect of the embodiments described herein relates to a service, such as an application marketplace that can provide multiple different extension applications for use within an instant messaging application according to one or more embodiments described herein. The service or application marketplace can present a browsable view of multiple different extension applications and instant messaging application plug-ins, and provide for downloading those extension applications to a client device to allow the client device to install one or more extension applications. FIG. 14A illustrates an example of a method of using such a service or application marketplace, and Fig. 14B An example of a user interface showing a messaging extension application marketplace. In one embodiment, Fig. 14B The application marketplace shown can be invoked from a browsable view of installed extension applications within the user interface of an instant messaging application on a client device. For example, selecting Figure 4B The icon 167 shown may cause the display Fig. 14B The user can then browse a collection of one or more messaging extension applications and select one or more extension applications that may be free or available for purchase. Fig. 14B In the example shown, the messaging extension app marketplace 725 may include a navigation bar 729 and a browsable view of one or more messaging extension apps, such as apps 726, 728, and 731 displayed on the touch screen 727. In one embodiment, the user can browse the apps by swiping the user's finger on the touch screen or using the navigation bar 729. The user can then choose to download and install one or more of the extension apps, and thereby, the instant messaging app on the user's client device can add icons representing the newly installed extension apps to the browsable view of installed extension apps, such as . Figure 4B In addition, the instant messaging application can add the installed extension application to the list of installed extension applications together with the identifier of the application provided by the application market ("application identifier"). Fig. 14B The App Store shown is one way to install extensions, but another way is to Fig.14A , where the installation process is initiated as a result of user interaction with a message in a message log of an instant messaging application.

[0107] See now Fig.14A, the instant messaging application in operation 701 may receive content (and optional resource locators and data) created by a remote device, such as an extension application on a first device, and may also receive an application identifier for the extension application on the remote device. In one embodiment, the application identifier may be the same identifier provided by the application market when the extension application is installed on the first device, or may be a different identifier associated with the identifier of the application store. Then in operation 703, the instant messaging application on the second device may display the content in a message bubble and, if provided, retain the resource locator and data. At this point, in this embodiment, the instant messaging application does not attempt to launch the extension application, which is not actually installed on the second device at this time. Then in operation 705, the instant messaging application receives a selection, such as a tap, of a message bubble containing content provided by the extension application on the first device, and the instant messaging application determines that the extension application (as identified by the received application identifier) ​​is not installed on the second device. At this point, if Fig.14A As shown in operation 707, the instant messaging application provides download and installation of the extension application on the second device and the user can choose to install the extension application identified by the application identifier in operation 701. In some cases, the user may need to purchase the extension application before downloading and installing the extension application. The provision of download and installation can be presented in a notification, such as Fig. 5F The notification 259 shown may include a selectable option to cause the download and installation of the extension application for the selected message bubble. Then in operation 709, the second device downloads and installs the extension application and may launch the newly installed extension application. In one embodiment, the download and installation may occur in the background while the instant messaging application remains in the foreground. In one embodiment, the progress of the download and installation may be shown in a progress bar, such as the progress bar 473 shown in the selected message bubble. After operation 709 is completed, the newly installed extension application may then be available on the second device, and the instant messaging application may provide the content and optional resource locator and data to the extension application on the second device in a view hosted by the instant messaging application in operation 711.

[0108] In one embodiment, the instant messaging application can cause installed extension applications to be automatically updated. In another embodiment, the instant messaging application can provide users with alerts or notifications that specific extension applications need to be updated, and in one embodiment, notifications about these updates can be received from the extension application market. This allows users to selectively decide whether to update a specific instant messaging application.

[0109] Another aspect of the embodiments described herein relates to a sticker extension application, and Fig.15 , Fig.16A , Fig. 16B and Fig. 16C An example of this is shown in Figure 1. The sticker extension application is used as an instant messaging application such as Figure 3A In one embodiment, the sticker application can be purchased from or otherwise obtained from a message extension application store (e.g., see FIG. 14B ), and the purchased sticker application can include a set of stickers. In one embodiment, the file size of the image is constrained to be less than a predetermined amount of data. In one embodiment, a user can create an image using a sticker extension application within the view of an instant messaging application, and then drag the image from the view of the sticker extension application in the message application to the message bubble in the record provided by the instant messaging application. In one embodiment, after dragging the image to the message bubble, the user can also scale (e.g., zoom in or out) and rotate the sticker image on the selected message bubble. In one embodiment, the user can also apply stickers to message bubbles that have been temporarily stored to be sent but have not yet been sent. In one embodiment, the sending device sends the encrypted image file to the message server and receives a download token (and an associated time to live (TTL) value), wherein the token represents the encrypted image file. As described herein, in one embodiment, the image files are encrypted on a per-sender basis, and thus the message server may store multiple instances of the same sticker, each encrypted with a different key for a different sending device and each with a time-to-live value that is refreshed when the appropriate sending device sends a download token (previously obtained from the server) to the server for a subsequent message. In one embodiment, the stickers may be animated.

[0110] When the sending device first sends the sticker image, which may be encrypted, to one or more messaging servers, it receives a download token and a time-to-live value. Fig.15 The message system 800 shown may be similar to Figure 2 Specifically, the group of message servers 801 may be similar to the message server 51, and the sending device 803 and the receiving device 805 may be similar to the client device 53 and the client device 54. In addition, in one embodiment, the sending device 803 and the receiving device 805 may use Figure 3A The architecture shown in FIG. 1 shows an instant messaging application 76, wherein the instant messaging application 76 is implemented by Figure 3A The IPC shown operates with a standalone extension application. When the sending device 803 first sends 811 a sticker image (which may be encrypted) together with a message (which may be encrypted) to a set of one or more message servers 801, the sending device 803 receives 812 a download token and a TTL value indicating when the encrypted sticker image is scheduled to expire on both the message server 801 and the sending device 803 (at the same time). If the sending device 803 sends 819 the sticker (the same sticker) in the message again to the same or different receiving device, and the TTL value has not expired, the sending device 803 will no longer upload the sticker image, but will send 819 the download token to the one or more message servers 801 which also causes the time to live value of the server 801 to be refreshed (at the same time causing the TTL value stored in the sending device 803 to be refreshed). If the time to live value expires, the sending device 803 will upload the encrypted sticker image again (which may be encrypted using a different key) and will receive another download token from the one or more message servers 801. The set of one or more servers 801 may maintain a database 802 of encrypted stickers having a corresponding token and a corresponding time to live value for each encrypted sticker. Specifically, encrypted sticker 1 may have a corresponding token value of token 1 as shown in row 821 of database 802, and a corresponding time to live value of TTL1. Similarly, an encrypted sticker image may have a copy 2 of the encrypted sticker in row 823 of its database 802, and corresponding token 2 and TTL2 in row 823. In an alternative embodiment, sticker 1 and sticker 2 may be encrypted images of the same sticker image, but are encrypted for different recipients (but from the same sending device) and using different keys, and are therefore stored separately in separate rows of the database. Although this article is relative to Fig.15 The described embodiment uses TTL values ​​for stickers, but it should be understood that other objects sent from instant messaging applications also use TTL values ​​in the same manner, but these other objects may include images, documents, executable files (e.g., games), etc.

[0111] Fig.16AA method according to one embodiment is shown, where a sending device 803 interacts with a set of one or more message servers 801. In operation 851, the sticker extension application creates a sticker image and a hash of the image, and may receive user input to determine image metadata, such as scale metadata and rotation metadata. In one embodiment, the creation of a sticker image may involve only selecting a specific sticker image, such as from Fig. 16C 835, and dragging a selected sticker, such as sticker 841, onto a message, such as a message bubble 839 (or any other message bubble in a message log or text scratch area) that can be sent by the communication device 831. Similarly, creation of a flower sticker on a message bubble 839 may simply involve dragging sticker 843 from the browsable sticker view 835 onto a message bubble 839 (or any other message bubble in a message log or text scratch area), and performing any image manipulations, such as scaling or rotating, while the flower is on the message bubble 839. See again Fig.16A In operation 851, the instant messaging application receives a sticker image and image metadata (e.g., rotation metadata and scaling metadata) for a location on a particular message bubble from a sticker application, as well as location metadata, and associates a bubble identifier (which can be identified as a particular message bubble) and a sticker extension application identifier with the sticker image. Then, in operation 853, the instant messaging application on the sending device uploads the encrypted message (if any) and the encrypted sticker image and metadata for delivery to the receiving device to one or more messaging servers. The upload process is as follows: Fig.15 Then, in operation 855, a sending device (such as sending device 803) receives a download token from one or more message servers 801 and stores the token and the associated TTL value of the corresponding encrypted sticker image. Fig.15812 of receiving. In operation 857 of FIG. 16A, the sending device may also optionally receive a confirmation of receipt of the message. Then, in operation 859 of subsequently sending the same image sticker to the same receiving device or other receiving device (in one embodiment), the sending device 803 may send 819 the associated token for the delivered sticker image to the same receiving device or other receiving device (in one embodiment), as long as the TTL value has not expired. In one embodiment, sending the token can refresh the TTL value at both the message server and the sending device 803. Therefore, the one or more message servers will refresh the corresponding time to live value in the database 802 when receiving the token sent for the second time and subsequently. In other words, if the time to live value has expired, the sending device 803 will upload another copy of the encrypted sticker image again, and receive a new download token and a new time to live value from one or more message servers 801.

[0112] Fig. 16B The method shown may be performed by the receiving device 805. In operation 875, the receiving device may receive the encrypted message along with the download token and corresponding metadata, and the process may be as follows: Fig.15 In one embodiment, the metadata may include a hash created by the remote sticker extension application and a sticker extension application identifier, and also include a bubble identifier and location metadata indicating a location on the identified message bubble, and scaling metadata and rotation metadata, which are used by the instant messaging application on device 805 to position, scale, and rotate the sticker. In operation 877, the instant messaging application determines whether the receiving device has a local copy of the sticker image stored on the receiving device based on the hash received in operation 875. If so, operation 881 is continued after operation 877, so that the process jumps to the following step. Fig. 16B If there is no local copy, then after operation 877, operation 879 is continued, and in operation 879, the receiving device 805 Fig.15The download token is sent to one or more messaging servers 801 in the illustrated transmission 815 to retrieve the encrypted sticker image, and then the instant messaging application decrypts the encrypted sticker image in operation 879. Then in operation 891, the receiving device uses the image metadata to generate a final sticker image that may include one or more rotations of the image and enlargement or reduction (or other modifications of the image); in one embodiment, the instant messaging application on the receiving device performs operation 891. Then in operation 893, the instant messaging application may display the final sticker image on a bubble in the message record (displayed on a bubble identified by a bubble identifier). Fig. 16C An example is shown where a user has created two sticker images and placed them on a message bubble 839 .

[0113] In one embodiment, the sticker extension application may provide a user interface that allows one or more users to assemble stickers from parts or portions of stickers in a library of sticker parts to create a final sticker that includes selected parts or portions from the library; the library may be provided by the sticker extension application. The sticker extension application may guide the user through the creation of the final sticker through a "wizard" that guides the user through the various stages involved in creating the final sticker. An example of such a sticker extension application is described in Appendix II of U.S. Provisional Patent Application No. 62 / 349,091, filed on June 12, 2016.

[0114] Fig.15The example shown is one embodiment of distributing or sending stickers to a receiving device, and two alternative embodiments will now be described. In a first alternative embodiment, when the developer of the sticker extension application uploads the extension application and its resources and assets (such as sticker images) to a distribution agency / service such as an online application store, the distribution agency / service can extract the assets and store them in a cloud-based storage system (e.g., one or more servers that support iCloud storage), and provide a unique identifier (such as a globally unique identifier (GUID)) for each sticker image, and associate the GUID with the corresponding sticker image. The distribution agency / service can also store the unique identifier through the sticker extension application (e.g., as part of a package that includes the sticker extension application) so that the sending device (e.g., device 803) can install the sticker extension application using the unique identifier for the sticker image included in the application. When the sending device uses the sticker created by the sticker extension application to send a message, the instant messaging application on the sending device sends the unique identifier of the sticker to the instant messaging application on the receiving device. The instant messaging application on the receiving device then retrieves the sticker image (if not stored locally on the receiving device) by sending the unique identifier to a cloud-based storage system, which responds by sending the identified sticker image to the receiving device.

[0115] A second alternative embodiment supports new stickers that can be generated by users (who use image creation or capture applications to generate new stickers), and the second alternative embodiment can also use a cloud-based storage service that uses an identifier of a sticker image (e.g., a hash of the sticker image) to retrieve sticker images when a receiving device requests stickers. When a sending device (e.g., sending device 803) sends a sticker, it can check a local storage device for a list of stickers previously sent to a set of one or more message servers and check whether the sticker has been sent before, and the sending device sends the sticker's identifier to the receiving device. The identifier can be a hash of the sticker image such as a SHA-1 hash or an MD1 hash, and the identifier will depend on the content of the sticker image and is a reliable and unique identifier of the sticker image in almost all cases. After receiving the identifier from the sending device, the receiving device can use the identifier to retrieve the sticker from the cloud-based storage service using the identifier, in one embodiment, the identifier is used as a key to find the sticker image in a database, and then the retrieved sticker image is sent to the receiving device based on the cloud storage service. When it checks the local storage of previously sent stickers, if the sending device determines that the sticker image has not been previously sent by the sending device's instant messaging application, the instant messaging application can send the sticker image to a cloud-based storage service (directly or through one or more messaging servers, which can create a hash (identifier) ​​of the sticker image and both are stored in the cloud-based storage service; the sending device also sends the identifier (e.g., the hash of the sticker image) to the receiving device, which can use the identifier to retrieve the sticker image from the cloud-based storage service).

[0116] In some embodiments, one or more application programming interfaces (APIs) may be used. An API is an interface implemented by a program code component or a hardware component (hereinafter referred to as an "API implementation component") that allows different program code components or hardware components (hereinafter referred to as "API calling components") to access and use one or more functions, methods, processes, data structures, categories, and / or other services provided by the API implementation component. The API may define one or more parameters transmitted between the API calling component and the API implementation component.

[0117] An API allows developers of API-calling components (which may be third-party developers) to utilize specified features provided by an API-implementing component. There may be one API-calling component or there may be more than one such component. An API may be a source code interface provided by a computer system or a library to support requests for services from an application. An operating system (OS) may have multiple APIs to allow applications running on the OS to call one or more of those APIs, and a service (such as a library) may have multiple APIs to allow applications using the service to call one or more of those APIs. An API may be specified in a programming language that can be interpreted or compiled when building an application.

[0118] In some embodiments, the API implementation component may provide more than one API, each providing a different view of the functionality implemented by the API implementation component or providing different aspects of the functionality implemented by the API implementation component. For example, one API of the API implementation component may provide a first set of functionality and may be exposed to third-party developers, and another API of the API implementation component may be hidden (not exposed) and provide a subset of the first set of functionality and also provide another set of functionality, such as testing or debugging functionality that is not in the first set of functionality. In other embodiments, the API implementation component itself may call one or more other components via the underlying API, and is therefore both an API calling component and an API implementation component.

[0119] The API defines the language and parameters used by the API calling component when accessing and using the specified features of the API implementation component. For example, the API calling component accesses the specified features of the API implementation component through one or more API calls or references exposed by the API (for example, implemented by function or method calls), and uses parameters to transmit data and control information via API calls or references. The API implementation component can return values ​​through the API in response to API calls from the API calling component. Although the API defines the syntax and results of API calls (for example, how to reference API calls and what operations can be performed by API calls), the API may not reveal how the API call completes the function specified by the API call. Various API calls are transmitted via one or more application programming interfaces between the call (API calling component) and the API implementation component. Transmitting API calls may include issuing, initiating, referencing, calling, receiving, returning or responding to function calls or messages; in other words, the transmission may describe actions through API calling components or API implementation components. Function calls or other references of the API can send or receive one or more parameters through parameter lists or other structures. A parameter may be a constant, a key, a data structure, an object, an object class, a variable, a data type, a pointer, an array, a list, or a pointer to a function or method or another way of referencing data or other items to be transferred via an API.

[0120] In addition, data types or categories can be provided by the API and implemented by the API implementation component. Therefore, the API calling component can declare variables, use pointers to such types or categories, use or instantiate constant values ​​of such types or categories by using definitions provided in the API.

[0121] Typically, API can be used to access the service or data provided by the API implementation component or to start the execution of the operation or calculation provided by the API implementation component. By way of example, the API implementation component and the API call component can be any one of an operating system, a library, a device driver, an API, an application program, or other modules (it should be understood that the API implementation component and the API call component can be modules of the same or different types to each other). In some cases, the API implementation component can be implemented at least in part in firmware, microcode or other hardware logic components. In some embodiments, API can allow client programs (for example, game center applications) to use the services provided by a software development kit (SDK) library. In other examples, applications or other client programs can use the API provided by the application framework. In these examples, applications or client programs can be incorporated into the function or method provided by the SDK and provided by the API by calling, or use the data type or object defined in the SDK and provided by the API. In these examples, the application framework can provide a main event loop for the program, and the program responds to the various events defined by the framework. API allows applications to use the application framework to specify events and respond to events. In some embodiments, API calls may report to an application the capabilities or states of a hardware device, including those related to aspects such as input capabilities and states, output capabilities and states, processing capabilities, power states, storage capacity and states, and communication capabilities, and the API may be implemented in part by firmware, microcode, or other low-level logic components that execute in part on the hardware components.

[0122] The API calling component may be a local component (i.e., on the same data processing system as the API implementing component) or a remote component (i.e., on a different data processing system than the API implementing component) that communicates with the API implementing component through an API via a network. It should be understood that an API implementing component may also act as an API calling component (i.e., it may make API calls to an API exposed by a different API implementing component), and an API calling component may also act as an API implementing component by implementing an API exposed to a different API calling component.

[0123] The API may allow multiple API-calling components written in different programming languages ​​to communicate with the API-implementing component (so that the API may include features for converting calls and returns between the API-implementing component and the API-calling component); however, the API may be implemented in terms of a specific programming language. In one embedding, the API-calling component may call APIs from different providers, such as one set of APIs from an OS provider and another set of APIs from a plug-in provider, and another set of APIs from another provider (e.g., a provider of a software library) or a creator of another set of APIs.

[0124] Fig.17 is a block diagram illustrating an exemplary API architecture that can be used in one embodiment of the present invention. Fig.17 As shown in , the API architecture 3200 includes an API implementation component 3210 (e.g., an operating system, library, device driver, API, application, software, or other module) that implements an API 3220. The API 3220 specifies one or more functions, methods, classes, objects, protocols, data structures, formats, and / or other features of the API implementation component that can be used by an API calling component 3230. The API 3220 may specify at least one calling convention that specifies how a function in the API implementation component receives parameters from an API calling component and how a function returns a result to an API calling component. The API calling component 3230 (e.g., an operating system, library, device driver, API, application, software, or other module) makes an API call through the API 3220 to access and use the features of the API implementation component 3210 specified by the API 3220. The API implementation component 3210 may return a value to the API calling component 3230 through the API 3220 in response to the API call.

[0125] It should be understood that the API implementation component 3210 may include additional functions, methods, classes, data structures, and / or other features that are not specified by the API 3220 and are not available to the API calling component 3230. It should be understood that the API calling component 3230 may be located on the same system as the API implementation component 3210, or the API implementation component 3210 may be located remotely and accessed using the API 3220 via a network. Fig.17 A single API calling component 3230 is shown interacting with API 3220 , but it should be understood that other API calling components different from API calling component 3230 that may be written in a different language (or the same language) may use API 3220 .

[0126] The API implementation component 3210, API 3220, and API calling component 3230 may be stored in a machine-readable medium (e.g., a computer-readable medium) including any mechanism for storing information in a machine-readable form (e.g., a computer or other data processing system). For example, the machine-readable medium includes a disk, an optical disk, a random access memory, a read-only memory, a flash memory device, etc.

[0127] exist Fig.18 ("software stack"), in one embodiment of the present invention, an application can use several service APIs to call service A or B, and use several OS APIs to call the operating system (OS). Services A and B can use several OS APIs to call the OS.

[0128] Note that Service 2 has two APIs, one of which (Service 2 API 1) receives a call from Application 1 and returns a value, and the other (Service 2 API 2) receives a call from Application 2 and returns a value. Service 1 (e.g., a software library) calls OS API 1 and receives a return value, and Service 2 (e.g., a software library) calls both OS API 1 and OS API 2 and receives a return value. Application 2 calls OS API 2 and receives a return value.

[0129] The systems and methods described herein can be implemented in a variety of data processing systems and devices including general-purpose computer systems, special-purpose computer systems, or hybrid systems of general-purpose and special-purpose computer systems. Exemplary data processing systems that can use any of the methods described herein include desktop computers, laptop computers, tablet computers, smart phones, cellular phones, personal digital assistants (PDAs), embedded electronic devices, or consumer electronic devices.

[0130] Fig.19 is a block diagram of data processing system hardware according to one embodiment. Fig.19 Various components of a data processing system that may be incorporated into a mobile device or handheld device are shown, but they are not intended to represent any particular configuration or manner of interconnecting these components, and such details are not germane to the present invention. Fig.19 Other types of data processing systems having fewer or more components than shown may be used with the present invention.

[0131] like Fig.19As shown in , the data processing system includes one or more buses 1309 for interconnecting various components of the system. As is well known in the art, one or more processors 1303 are coupled to one or more buses 1309. Memory 1305 may be DRAM or non-volatile RAM, or may be flash memory, or other types of memory, or a combination of such storage devices. The memory is coupled to one or more buses 1309 using techniques known in the art. The data processing system may also include non-volatile memory 1307, which may be a hard drive, or flash memory, or magnetic optical drive, or magnetic storage, or optical drive, or other types of memory systems that retain data even after the system is powered off. Both non-volatile memory 1307 and memory 1305 are coupled to one or more buses 1309 using known interfaces and connection techniques. Display controller 1322 is coupled to one or more buses 1309 to receive display data to be displayed on display device 1323. Display device 1323 may include an integrated touch input for providing a touch screen. The data processing system may also include one or more input / output (I / O) controllers 1315 that provide an interface for one or more I / O devices, such as one or more mice, touch screens, touch pads, joysticks, and other input devices (including those I / O devices known in the art), and output devices (e.g., speakers). As is known in the art, input / output devices 1317 are coupled through one or more I / O controllers 1315.

[0132] Although Fig.19 It is shown that non-volatile memory 1307 and memory 1305 are coupled to one or more buses directly rather than through a network interface, but it should be understood that the present invention can utilize non-volatile memory away from the system, such as a network storage device coupled to the data processing system through a network interface such as a modem or an Ethernet interface. As is well known in the art, the bus 1309 can be connected to each other through various bridges, controllers and / or adapters. In one embodiment, the I / O controller 1315 includes one or more of a USB (Universal Serial Bus) adapter for controlling USB peripherals, an IEEE 1394 controller for IEEE1394 compatible peripherals, or a Thunderbolt controller for controlling Thunderbolt peripherals. In one embodiment, one or more network devices 1325 can be coupled to one or more buses 1309. The network device 1325 can be a wired network device (e.g., Ethernet) or a wireless network device (e.g., WI-FI, Bluetooth).

[0133] It will be apparent from this description that aspects of the present invention may be embodied at least in part in software. That is, the technology may be executed in a data processing system in response to its processor executing a series of instructions contained in a storage medium such as a non-transitory machine-readable storage medium (e.g., DRAM or flash memory). In various examples, hard-wired circuits may be used in combination with software instructions to implement the present invention. Therefore, the technology is not limited to any specific combination of hardware circuitry and software, or to any specific source of instructions executed by a data processing system. In addition, it should be understood that when describing a mobile device or handheld device, this specification covers mobile devices (e.g., laptop devices, platform devices), handheld devices (e.g., smart phones), and embedded systems suitable for wearable electronic devices.

[0134] The present disclosure recognizes that the use of personal information data in the present technology can be used to benefit users. For example, the personal information data can be used to deliver information or targeted content that users are more interested in. Therefore, the use of such personal information data can enable planned control of the delivered content. In addition, the present disclosure also contemplates other uses of personal information data that are beneficial to users.

[0135] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other uses of such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of the entity and not shared or sold outside these legal uses. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any necessary steps to safeguard and protect access to such personal information data, and ensure that others who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0136] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to health information or advertising delivery services, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. As another example, a user may choose not to provide location information for a target content delivery service. As another example, a user may choose not to provide precise location information, but permit the transmission of location area information.

[0137] In the foregoing description, specific exemplary embodiments have been described. It will be apparent that various modifications may be made to those embodiments without departing from the broader spirit and scope as set forth in the following claims. Accordingly, the description and drawings should be regarded as being for illustrative purposes rather than for restrictive purposes.

[0138] appendix

[0139] The following appendix describes an example of an application programming interface for an iOS device using the iOS operating system from Apple Inc. (Cupertino, California). The API is used to interact between the extension application described herein and an instant messaging application on an iOS device.

[0140] Summary of changes to the iMessage API for extending apps

[0141] Some of the defined types are renamed, some protocols are changed to classes and vice versa. The affected types are:

[0142] ο Turned MSMessageContext into a class and renamed it MSConversation. This name more accurately describes what the class represents.

[0143] o Renamed MSMessagePayload to MSMessage. The new name better describes the class, as this object represents more than just a message payload that will be sent.

[0144] ο Rename MSBalloonLayout to MSMessageLayout. The word balloon is used to describe the current UI and should not be loaded into the API.

[0145] οRename MSBalloonTemplateLayout to MSMessageTemplateLayout.

[0146] o Convert MSMessageTemplateLayout to a protocol and add a class with the same name to implement the protocol. Developers using the API are confused when they cannot create their own objects that implement the MSBalloonLayout protocol. This change will allow developers to provide their own MSMessageTemplateLayout conforming objects.

[0147] οConvert the MSSticker protocol into a class. Stickers are similar to the concept of UIImage, so they should also be specific classes. The relationship between MSSticker and MSStickerView should be similar to the relationship between UIImage and UIImageView.

[0148] New features:

[0149] o New MSSession class. AMSMessages initialized with a session can participate in "breadcrumb" behavior. That is, all messages created in a session with the same session will update a single balloon in the session record, and optionally leave a trail of breadcrumb entries in the record. Prior to this proposal, this behavior was implemented by modifying the input message provided by the MSMessageContext and passing the modified object to -updateMessagePayload:completionHandler:.

[0150] o Added the MSConversationDelegate protocol, which defines the delegate interface for the MSConversation delegate. This delegate will receive callbacks when the user taps the send button while an MSMessage is already stored for sending and when the user deletes the stored MSMessage from the input field. This delegate will receive callbacks if a new message from the extended application type arrives in the conversation while the extended application is active.

[0151] o Added a category for NSExtensionContext that allows extension apps to get information about how the presentation is presented and request changes to the presentation style. This enables extension apps to request to expand or contract the presentation and request to cancel it entirely.

[0152] o Replace the callbacks on MSMessagePayloadProvider with a set of callbacks modeled in UIApplicationDelegate.

[0153] o Renamed -updateMessagePayload:completionHandler: to -insertMessage:localizedChangeDescription:completionHandler: The word update seems confusing as this method will insert a new message when there is no stored MSMessage to send. This will also add a localizedChangeDescription parameter that allows developers to provide breadcrumb text.

[0154] Detailed Design

[0155] Updates to headers

[0156] Update MSMessagePayloadProvider.h

[0157]

[0158]

[0159]

[0160]

[0161]

[0162] Add MSMessage.h as a replacement for MSMessagePayload.h

[0163]

[0164]

[0165]

[0166] Add MSMessageLayout.h to replace MSBalloonLayout.h

[0167]

[0168] Add MSMessageTemplateLayout.h to replace MSBalloonTemplateLayout.h

[0169]

[0170]

[0171] Add MSSession.h

[0172]

[0173] Update MSSticker.h

[0174]

[0175]

[0176] Update MSStickerBrowserView.h

[0177]

[0178]

[0179] Update MSStickerBrowserViewController.h

[0180]

[0181] Update MSStickerBrowserViewDataSource.h

[0182]

[0183]

[0184] Update MSStickerView.h

[0185]

[0186] Examples of use

[0187] Send a simple message

[0188]

[0189]

[0190] Sending a conversation message

[0191]

[0192] Send Text

[0193]

[0194] Send stickers

[0195]

Claims

1. A method for operating in conjunction with a messaging application, the method comprising: Determining, by the message application, a type of the extension application in response to receiving a message containing content created by the extension application; When the extension application is an extension application of the first type, automatically launching the extension application in response to displaying a message bubble corresponding to the content in a message record of the message application; as well as When the extension app is an extension app of the second type, launching of the extension app is postponed until a selection of the message bubble displayed in the message record of the messaging app is received.

2. The method of claim 1, wherein the selection is received from a user selecting the message bubble in the record, and wherein the type of the extension application is determined based at least in part on the message. 3 . The method of claim 1 , wherein the extension application specifies whether it operates as a first type or a second type of extension.

4. The method of claim 1 , wherein the messaging application is a text messaging application that sends text messages to another messaging application through one or more servers, and wherein an extension application of the first type is launched upon receipt but remains in the background until a user causes the extension application to be displayed in place of an on-screen keyboard. 5 . The method of claim 4 , wherein the extension app updates content displayed in one or more message bubbles in the message history while executing in the background.

6. The method of claim 1, wherein content from an extension app of a first type appears in the message bubble even when the extension app of the first type is not launched, and wherein after the extension app of the first type is launched, content from the extension app is updated through inter-process communication between the extension app and the message app, and wherein the message app operates in a first sandbox process and the extension app operates in a second sandbox process different from the first sandbox process, and wherein the extension app receives content through the message app on a first device, the content being created by another extension app that sends the content through another message app on a second device, and wherein the message app determines whether the extension app is of the first type or the second type.

7. A device comprising: Memory; and At least one processor configured to: Determining, by the message application, a type of the extension application in response to receiving a message containing content created by the extension application; When the extension application is an extension application of the first type, automatically launching the extension application in response to displaying a message bubble corresponding to the content in a message record of the message application; as well as When the extension app is an extension app of the second type, launching of the extension app is postponed until a selection of the message bubble displayed in the message record of the messaging app is received.

8. The apparatus of claim 7, wherein the selection is received from a user selecting the message bubble in the recording.

9. The device of claim 7, wherein the extension application specifies whether it operates as a first type or a second type of extension.

10. The device of claim 7, wherein the messaging application is a text messaging application that sends text messages to another messaging application through one or more servers, and wherein the first type of extension application is launched upon receipt but remains in the background until a user causes the extension application to be displayed in place of an on-screen keyboard. 11 . The device of claim 10 , wherein the extension app updates content displayed in one or more message bubbles in the message history while executing in the background.

12. The device of claim 7, wherein content from the extension app of the first type appears in the message bubble even when the extension app of the first type is not launched, and wherein after the extension app of the first type is launched, the content from the extension app is updated through inter-process communication between the extension app and the message app, and wherein the message app operates in a first sandbox process and the extension app operates in a second sandbox process different from the first sandbox process, and wherein the extension app receives content through the message app on the first device, the content being created by another extension app that sends the content through another message app on the second device, and wherein the message app determines whether the extension app is of the first type or the second type.

13. A method comprising: Displaying a first message bubble containing first content created by the first extension application or associated with the first content in a message history of the first message application, the first message bubble having a session identifier maintained for the first message application; receiving, at a first messaging application on the first device, second content for a session occurring between the first extension application and a second extension application on the second device, the session being identified by the session identifier; The first content is converted into a breadcrumb and a second message bubble containing or associated with the second content is displayed, wherein the first messaging application associates the conversation identifier with both the breadcrumb and the second message bubble. The method of claim 13 , wherein the first extension application provides third content to be displayed as at least a portion of the breadcrumbs. 15 . The method of claim 13 , wherein the second content is created by a second extension application on the second device and is sent to the first device through a second message application on the second device. The method of claim 13 , wherein the first messaging application converts the first content into the breadcrumbs.

17. The method of claim 16, wherein the first content is created before the second content is received, and wherein the second content is an update or modification of the first content.

18. The method of claim 17, wherein the message log includes a plurality of message bubbles identified with the conversation identifier, and wherein at least a subset of the plurality of message bubbles are converted into the breadcrumbs.

19. A device comprising: Memory; and At least one processor configured to: Displaying a first message bubble containing first content created by the first extension application or associated with the first content in a message history of the first message application, the first message bubble having a session identifier maintained for the first message application; receiving, at a first messaging application on the first device, second content for a session occurring between the first extension application and a second extension application on the second device, the session being identified by the session identifier; The first content is converted into a breadcrumb and a second message bubble containing or associated with the second content is displayed, wherein the first messaging application associates the conversation identifier with both the breadcrumb and the second message bubble. 20 . The device of claim 19 , wherein the first extension application provides third content to be displayed as at least a portion of the breadcrumbs. 21 . The device of claim 19 , wherein the second content is created by a second extension application on the second device and is sent to the first device through a second message application on the second device.

22. The device of claim 19, wherein the first messaging application converts the first content into the breadcrumbs.

23. The apparatus of claim 22, wherein the first content is created before the second content is received, and wherein the second content is an update or modification of the first content.

24. The device of claim 23, wherein the message log includes a plurality of message bubbles identified with the conversation identifier, and wherein at least a subset of the plurality of message bubbles are converted into the breadcrumbs.

Images