Generating segment packets based on selection of portions of web page

Abstract

Systems and methods for segment packet generation may include obtaining input data (e.g., input data describing a gesture). Input data may be processed to determine a content item selected by the input. A segment packet may be generated based on the content item, the segment packet may include the content item, address data, and location data. The segment packets may be configured to be available for interaction to navigate to a source web page of the content item, including to navigate to a particular portion of the web page that includes the content item.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of U.S. Non-Provisional Patent Application No. 18 / 081,814, filed on December 15, 2022, which claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 344,783, filed on May 23, 2022. U.S. Non-Provisional Patent Application No. 18 / 081,814 and U.S. Provisional Patent Application No. 63 / 344,783 are hereby incorporated by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to generating interactive snippet groupings in response to user input. More specifically, the present disclosure relates to obtaining user input to select a content item to be saved with a snippet grouping that can later be selected to provide a portion of a web page that includes the content item. Background Art

[0004] Saving text, images, and / or audio from a web page may allow a user to re-experience the text, images, and / or audio locally without having to connect to the Internet. However, the saving process may provide limited context to the source of the data, and in instances where a user wishes to view the context of the saved data, the user must use the data as a search query, navigate their browsing history, or try to remember how they originally entered the web page. Additionally, when the source of the web page is found, the user may still need to examine a large portion of the web page to find exactly where in the web page the saved data originally originated. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computing system. The system may include: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include providing data describing a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items. The operations may include obtaining input data. The input data may include a request to save one or more content items from the multiple content items. The operations may include generating a fragment grouping. In some implementations, the fragment grouping may include one or more content items. The fragment grouping may include address data. The address data may describe the URL of the web page. The fragment grouping may include location data. The location data may describe the location of one or more content items within the web page. The operations may include storing the fragment grouping. The fragment grouping may be associated with a specific user.

[0007] In some implementations, the operations may include receiving a fragment request to provide a fragment interface. The fragment interface may include interactive elements associated with the fragment grouping. The operations may include providing the fragment interface for display, receiving an interface selection to select the interactive element, and providing a portion of a web page for display. The portion of the web page may include the location of one or more content items within the web page. In some implementations, the operations may include receiving an insert input. The insert input may include user input requesting that one or more content items be inserted into a different interface. The operations may include providing the fragment grouping to a third-party server computing system.

[0008] In some implementations, generating the segment grouping may include obtaining one or more content items and generating a graphics card. The graphics card may describe the one or more content items. Generating the segment grouping may include storing the graphics card as a graphical representation of the segment grouping. In some implementations, the position data may include at least one of a scroll position, a start node, or an end node. The scroll position may describe the position of the one or more content items relative to other portions of the web page. The start node may describe the position where the one or more content items begin. The end node may describe the position where the one or more content items end.

[0009] In some implementations, the one or more content items may include at least one of an image, a video, a graphical depiction of a product, or audio. The operations may include processing the one or more content items to determine entities associated with the one or more content items and generating entity tags based on the entities. The segment groupings may include the entity tags. In some implementations, storing the segment groupings may include storing the segment groupings locally on the mobile computing device.

[0010] Another example aspect of the present disclosure relates to a computer-implemented method. The method may include obtaining input data by a computing system including one or more processors. The input data may describe a selection of a content item associated with a segment grouping. The method may include obtaining address data and position data associated with the segment grouping by the computing system. The address data may be associated with a web page. The content item may be associated with the web page. In some implementations, the position data may describe a location of the content item within the web page. The method may include obtaining web page data by the computing system. The web page data may be obtained based at least in part on the address data. The method may include determining, by the computing system, a location within the web page associated with the content item and providing, by the computing system, a portion of the web page. The portion of the web page may include the location of the content item.

[0011] In some implementations, the address data may include a uniform resource locator. The location data may include a text snippet. Determining, by the computing system, a location within a web page associated with the content item may include adding, by the computing system, the text snippet to the uniform resource locator to generate a shortcut link, and inputting, by the computing system, the shortcut link into a browser.

[0012] In some implementations, the segment grouping may be generated by providing a graphical user interface by a computing system. The graphical user interface may include a graphical window for displaying a web page. The web page may include multiple content items. The segment grouping may be generated by obtaining selection data by a computing system; generating the segment grouping by a computing system; and storing the segment grouping by the computing system in a user database. The selection data may include a request to save a content item from a plurality of content items. In some implementations, providing a portion of a web page by a computing system may include providing one or more indicators to the portion of the web page. The one or more indicators may indicate a content item associated with the segment grouping. The one or more indicators may include highlighted text associated with the content item. In some implementations, the segment grouping may be associated with a user account of a particular user. The user account may be associated with one or more platforms.

[0013] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause one or more computing devices to perform operations. These operations may include providing a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items. These operations may include receiving gesture data. In some implementations, the gesture data may describe a gesture associated with a portion of the web page. These operations may include processing the gesture data to determine a selected content item. The selected content item may be associated with a portion of the web page. These operations may include generating a fragment grouping based on the gesture data. In some implementations, the fragment grouping may include the selected content item.

[0014] In some implementations, the segment grouping may include address data and location data. The address data may be associated with a web page. The location data may describe the location of the selected content item within the web page. The gesture may include a circular gesture that surrounds a portion of the web page. In some implementations, processing the gesture data to determine the selected content item may include: determining the portion of the web page surrounded by the gesture; determining the focus of the portion; and determining that the selected content item is associated with the focus of the portion. In some implementations, processing the gesture data to determine the selected content item may include processing the gesture data with a machine learning model to determine the selected content item. The gesture data may describe a touch input to a touch screen display of a mobile computing device.

[0015] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0016] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0018] Figure 1A A block diagram of an example computing system that performs fragment grouping generation according to an example embodiment of the present disclosure is depicted.

[0019] Figure 1B A block diagram of an example computing device performing fragment grouping generation is depicted according to an example embodiment of the present disclosure.

[0020] Figure 1C A block diagram of an example computing device performing fragment grouping generation is depicted according to an example embodiment of the present disclosure.

[0021] Figure 2 Depicted is a diagram of an example segment grouping generation interface according to an example embodiment of the present disclosure.

[0022] Figure 3 Depicted are diagrams of example gesture interactions according to example embodiments of the present disclosure.

[0023] Figure 4 Depicted is a diagram of an example segment grouping generation interface according to an example embodiment of the present disclosure.

[0024] Figure 5Depicted is a diagram of an example snippet grouping generation and favorites adding interface according to an example embodiment of the present disclosure.

[0025] Figure 6 Depicted is a flow chart of an example method for performing fragment grouping generation according to an example embodiment of the present disclosure.

[0026] Figure 7 Depicted is a flow diagram of an example method for performing segment grouping interactions according to an example embodiment of the present disclosure.

[0027] Figure 8 Depicted is a flow diagram of an example method for performing segment grouping generation based on gesture input according to an example embodiment of the present disclosure.

[0028] Fig. 9 Depicted is a diagram of an example favorites adding interface according to an example embodiment of the present disclosure.

[0029] Fig. 10A Depicted is a diagram of an example segment grouping interaction according to an example embodiment of the present disclosure.

[0030] Fig. 10B Depicted is a diagram of an example segment grouping search according to an example embodiment of the present disclosure.

[0031] Fig. 10C Depicted is a diagram of an example highlight interface according to an example embodiment of the present disclosure.

[0032] Fig.11 A diagram of an example graphics card is depicted in accordance with an example embodiment of the present disclosure.

[0033] Fig.12 Depicted is a diagram of an example summary snippet grouping generation interface according to an example embodiment of the present disclosure.

[0034] Fig.13 Depicted is a diagram of an example snippet grouping suggestion interface according to an example embodiment of the present disclosure.

[0035] Fig.14 Depicted is a diagram of an example snippet grouping generation and sharing interaction according to an example embodiment of the present disclosure.

[0036] Fig.15 Depicted is a diagram of an example graphics card customization interface according to an example embodiment of the present disclosure.

[0037] Fig.16 Depicted is a block diagram of an example segment grouping generation system according to an example embodiment of the present disclosure.

[0038] Reference numerals that are repeated across multiple drawings are intended to identify like features in various implementations. DETAILED DESCRIPTION

[0039] Overview

[0040] In general, the present disclosure relates to generating interactive snippet groupings in response to user input. More specifically, the present disclosure relates to obtaining user input to select content items to be saved with a snippet grouping, which can be selected later to provide a portion of a web page including the content items. For example, a user can select a portion of a web page and / or a data file. Data describing the selected portion can then be stored together with information associated with the web page / data file and the location of the portion relative to the web page / data file. The stored data set can be a snippet grouping including a graphical representation of the selected portion, which can include a graphic card with text and / or images from the selected portion. The snippet grouping can be stored for later reference and / or can be shared by other users. The snippet grouping can enable a user to view the selected portion and then navigate to a specific location of the selected portion in the original web page / data file after selection. These systems and methods may include providing a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items. These systems and methods may include obtaining input data. The input data may include a request to save one or more content items in the multiple content items. The snippet grouping can be generated. The segment grouping may include one or more content items, address data, and location data. In some implementations, the address data may describe the URL of the web page. The location data may describe the location of the one or more content items within the web page. The segment grouping may be stored in a user database.

[0041] For example, the systems and methods disclosed herein may include providing a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items. The displayed web page may include text, one or more images, one or more interactive user interface elements, one or more videos, and / or one or more audio clips. The graphical user interface may be part of a browser application.

[0042] These systems and methods can obtain input data. The input data may include a request to save one or more content items in a plurality of content items. In some implementations, the one or more content items may include at least one of an image, a video, a graphical description of a product, or an audio. The graphical user interface may be updated to include one or more user interface elements for interacting with the one or more content items, which may include saving the one or more content items and may include different options for formatting the saving. Alternatively and / or in addition, the overlay interface may provide a pop-up interface in response to the request. The input data may describe a gesture associated with the one or more content items (e.g., a circle around the one or more content items).

[0043] The snippet groupings may be generated. The snippet groupings may be generated based on the input data. The snippet groupings may include one or more content items, address data, and location data. Generating the snippet groupings may include processing the one or more content items with one or more machine learning models to generate a semantically understood output that may be used for summarization, annotation, and / or classification.

[0044] One or more content items may include text data (e.g., text data describing words, sentences, quotations, and / or paragraphs), image data (e.g., images of objects (e.g., products)), video data (e.g., videos and / or one or more frames of videos), audio data (e.g., waveform data), and / or potentially encoded data. One or more content items may include multimodal data. Address data may include a resource locator (e.g., a web address) and / or a file address associated with a web page. Position data may describe the location of one or more content items on a web page and / or file. For example, position data may indicate the beginning and end of one or more content items in a web page and / or file.

[0045] In some implementations, generating a segment grouping may include obtaining one or more content items and generating a graphics card. The graphics card may describe one or more content items. The graphics card may include text data overlaid on a color and / or image. The color may be determined based on the dominant color of the web page. In some implementations, the color may be predetermined and / or may be determined based on the surrounding content items. The image may be an image determined based on the determined theme of the content item. Alternatively and / or in addition, the image may be an image close to the content item. In some implementations, the graphics card may include a font determined based on a font used in the web page. The graphics card may include a background, text describing the content item (e.g., the content item and / or a summary of the content item), and / or text and / or a logo describing the source and / or the determined entity. In some implementations, the text size of the text in the graphics card may be based on the amount of text in the content item.

[0046] The address data may describe the web address of the web page. The address data may include a uniform resource identifier and / or a uniform resource locator. In some implementations, the address data may include data describing the source of the content item.

[0047] The position data may describe the position of one or more content items within a web page. In some implementations, the position data may include at least one of a scroll position, a start node, or an end node. The scroll position may describe the position of one or more content items relative to other parts of the web page. In some implementations, the start node may describe the position where one or more content items start. The end node may describe the position where one or more content items end. The position data may include a text fragment ("Scroll to Text Fragment" published by Tomayac et al. (May 20, 2022 at 9:40 p.m.), GITHUB, https: / / github.com / WICG / scroll-to-text-fragment), which may be used to indicate the position of the content item. In some implementations, the text fragment may include one or more text instructions associated with the position. The text instruction may include a string of code start data (e.g., the first text associated with the content item and / or the first pixel associated with the content item) and / or end data (e.g., the last text associated with the content item and / or the last pixel associated with the content item). The text instructions may be used to search a web page for a data set that matches the beginning and / or end of a content item.

[0048] In some implementations, these systems and methods may include processing one or more content items to determine entities associated with the one or more content items. Entity tags may be generated based on the entities. The segment groupings may include the entity tags.

[0049] The fragment groupings may be stored in a user database. In some implementations, storing the fragment groupings in the user database may include storing the fragment groupings locally on the mobile computing device. Additionally and / or alternatively, the graphics card may be stored as a graphical representation of the fragment groupings. The graphics card may be automatically generated and may be customizable by the user. The graphics card may start from a template that may be customized based on other content in the web page, based on user input, and / or based on other contexts. In some implementations, the graphics card may include multimodal data. Alternatively and / or additionally, the fragment groupings may be stored on a server computing system. In some implementations, if a content item references another content item, these systems and methods may obtain additional content items and save the additional content items in the fragment groupings.

[0050] In some implementations, these systems and methods may include receiving a fragment request to provide a fragment interface. The fragment interface may include an interactive element associated with the fragment grouping. The fragment interface may be provided for display. An interface selection may be received, and the interface selection may describe a selection to select the interactive element. A portion of a web page may then be provided for display. In some implementations, the portion of the web page may include a location of one or more content items within the web page.

[0051] In some implementations, these systems and methods may include receiving an insert input. The insert input may include a user input requesting that the content item be inserted into a different interface. The segment grouping may be provided to a third party server computing system.

[0052] Additionally and / or alternatively, these systems and methods may include adding the segment grouping to favorites. The segment grouping may be added to the favorites based on received user input. Alternatively and / or additionally, the segment grouping may be automatically added to the favorites based on a determined entity associated with the content item. In some implementations, the segment grouping may be added to the favorites based on a source of the content item (e.g., a type of webpage, a type of media provider, and / or based on a type of content item).

[0053] In some implementations, segment groupings can be generated based on content items obtained from a source other than a web page, such as a mobile application, a large data file (eg, a downloaded video or book), and / or another data source.

[0054] These systems and methods may include providing a specific portion of a web page for display in response to an interaction with a segment grouping. For example, these systems and methods may include obtaining input data. The input data may describe a selection of a content item associated with the segment grouping. Address data and position data associated with the segment grouping may be obtained. The address data may be associated with the web page. The content item may be associated with the web page. Additionally and / or alternatively, the position data may describe the location of the content item within the web page. These systems and methods may obtain web page data. The web page data may be obtained based at least in part on the address data. The location within the web page associated with the content item may be determined. A portion of the web page may then be provided. The portion of the web page may include the location of the content item.

[0055] These systems and methods may obtain input data. In some implementations, the input data may describe a selection of content items associated with a segment grouping. The segment grouping may be associated with a user account of a particular user. Additionally and / or alternatively, the user account may be associated with one or more platforms. The segment grouping may include content items and deep links. Alternatively and / or additionally, the segment grouping may include segments (e.g., content items and / or media data generated based on the content items (e.g., a summary of a graphics card and / or content items)), address data (e.g., a uniform resource locator and / or a uniform resource identifier), and / or metadata. The segment grouping may include location data (e.g., metadata indicating the location of the content item within a web page, a text segment for identifying start data and end data, one or more pointers, and / or one or more scroll positioning data).

[0056] In some implementations, the segment grouping may be generated by providing a graphical user interface for display. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include a plurality of content items. The segment grouping generation may include obtaining selection data. The selection data may include a request to save a content item from the plurality of content items. In some implementations, the segment grouping generation may include generating the segment grouping and storing the segment grouping in a user database.

[0057] These systems and methods can obtain address data and position data associated with the segment grouping based on the input data. The address data can be associated with the web page. The content item can be associated with the web page. Additionally and / or alternatively, the position data can describe the location of the content item within the web page.

[0058] The web page data may then be obtained. The web page data may be obtained based at least in part on the address data (e.g., by navigating to the web page using a uniform resource locator). Alternatively and / or in addition, a file may be obtained based on the address data (e.g., the address data may describe a file location and may be used to obtain the particular file).

[0059] A location within a web page (and / or file) associated with the content item may be determined based on the obtained location data. The location may be determined based on a text snippet, one or more pointers, and / or via web page processing.

[0060] In some implementations, the address data may include a uniform resource locator. Additionally and / or alternatively, the location data may include one or more text snippets. Then, determining a location within the web page associated with the content item may include adding the text snippet to the uniform resource locator to generate a shortcut link and inputting the shortcut link into the browser.

[0061] The portion of the web page may then be provided for display. The portion of the web page may include the location of the content item. In some implementations, providing the portion of the web page may include providing one or more indicators to the portion of the web page. The one or more indicators may indicate the content item associated with the segment grouping. In some implementations, the one or more indicators may include highlighted text related to the content item.

[0062] These systems and methods may include processing gesture data. For example, these systems and methods may include providing a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items. These systems and methods may receive gesture data. The gesture data may describe a gesture associated with a portion of the web page. The gesture data may be processed to determine a selected content item. The selected content item may be associated with a portion of the web page. A segment grouping may be generated based on the gesture data. In some implementations, the segment grouping may include the selected content item.

[0063] Specifically, a graphical user interface may be provided for display. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items. The web page may be associated with a uniform resource locator and / or a source code. Multiple content items may include structured text data (e.g., a body paragraph and / or one or more titles), a space, one or more images, and / or an audio content item.

[0064] Then, gesture data may be received. The gesture data may describe a gesture associated with the portion of the web page. In some implementations, the gesture may include a circular gesture that encompasses the portion of the web page. The gesture data may describe a touch input to a touch screen display of the mobile computing device.

[0065] The gesture data may be processed to determine the selected content item. The selected content item may be associated with a portion of a web page. For example, a gesture may encompass an image and one or more lines of text in a web page that includes multiple lines and / or multiple images.

[0066] In some implementations, processing the gesture data to determine the selected content item may include: determining a portion of the webpage encompassed by the gesture; determining a focus of the portion; and determining that the selected content item is associated with the focus of the portion.

[0067] Alternatively and / or in addition, processing the gesture data to determine the selected content item may include processing the gesture data with a machine learning model to determine the selected content item. The machine learning model may be trained to determine the start and end of the selected content item based on proximity to gesture boundaries, grammar, structural data, whitespace, and / or semantic cohesion.

[0068] In some implementations, the selected content item may be determined based on a determined matching area associated with a rectangle determined based on a circle. The rectangle may be a rectangle of data composed based on the syntax of the web page. In some implementations, the selected content item may be determined based on determined word boundaries and / or based on determined media content boundaries. The determination may be based on hypertext markup language code boundaries. In some implementations, the source code of the web page may be parsed, and the parsed data may be processed.

[0069] Alternatively and / or in addition, the determination may include calculating the area of ​​a gesture rectangle associated with the gesture. The area of ​​one or more content item elements may be determined. The intersection area between the area of ​​the gesture rectangle and each of the areas of different content item elements may be determined. The element with the largest intersection may have the highest probability of selection and may therefore be determined to be the selected content item.

[0070] A segment grouping may then be generated based on the gesture data. The segment grouping may include the selected content item. In some implementations, the segment grouping may include address data and location data. The address data may be associated with a web page, and the location data may describe the location of the content item within the web page.

[0071] A save interface may then be provided for display. The save interface may provide an interactive interface element that may be selected to add the segment grouping to a favorite. A favorites interface may then be provided. A drag input may then be received that drags a graphical representation of the segment grouping to a graphical tile that depicts a particular favorite. The segment grouping may then be stored in the favorites (e.g., the segment grouping may be stored along with a relationship tag that links the segment grouping to a particular favorite). In some implementations, the graphical representation may change size and / or scale when the graphical representation of the segment grouping is dragged to a particular favorite. The size and scale changes may provide an intuitive indication of the addition to the favorite while providing an aesthetically pleasing display.

[0072] In some implementations, content items may be processed to determine entities associated with the content items. Entities may be determined by processing content items (e.g., text data, image data, audio data, video data, latent coding data, and / or link data) with a machine learning model (e.g., an image classification model, an object classification model, a text classification model (e.g., a natural language processing model), a segmentation model, a semantic model, and / or a detection model) to generate entity data (e.g., a classification). Relationship data may then be generated based on the entity data and added to the segment groupings. The relationship data may include one or more entity tags and / or references to other related segment groupings and / or related web pages or content items.

[0073] The snippet groupings may be searchable within one or more applications and / or databases. Additionally and / or alternatively, the snippet groupings may be shared via a messaging application, a social media application, and / or via another application.

[0074] In some implementations, content items may be processed with one or more machine learning models to generate tags that may be stored in the segment groups. The tags may then be used as searchable tags to display the segment groups in response to a search query. Tags may be determined based on the content of the content item (e.g., recognized words, recognized objects in an image, characteristics of a video frame or audio stream, etc.).

[0075] In some implementations, segment group generation occurs after selecting a segment group generation interface element, which can open a segment group generation interface. Alternatively and / or additionally, a user can select a content item and one or more (e.g., two) pop-ups can be provided with various options for interacting with the content item, and one of the options can include segment group generation. Alternatively and / or additionally, search results associated with the content item can be provided.

[0076] In some implementations, a screenshot request may be received. A screenshot may be generated and uploaded to a new client, and a word-gram may be generated. The word-gram may be used to receive data associated with the screenshot. The word-gram, the screenshot, and the screenshot details may be used to generate a snippet grouping.

[0077] These systems and methods can be implemented to generate fragment groups based on other data sources other than web pages. For example, these systems and methods can be used to generate fragment groups based on content items in data files stored locally and / or stored on a server computing system. The generated fragment groups may include content items (and / or graphics cards), address data, and position data. The address data may describe the storage location of the data file (e.g., the name of the drive and the name of any folder (e.g., G:\ResearchPapers\Quantum\Spin)). The position data may describe the location of the content item in the data file. The position data may include start data and end data, which can be used to find matching data in the data file, and then navigate to and highlight the matching data. Alternatively and / or in addition, the position data may include one or more pointers.

[0078] These systems and methods may enable a user to select subsets, excerpts, and / or portions of an object (e.g., text, images, and / or videos that may be part of a larger web page). In some implementations, these systems and methods may segment portions of text from a larger body of text, may segment portions of an image, may isolate frames in a video, and / or may segment portions of an audio file.

[0079] In some implementations, these systems and methods can be used as an extension in a browser application and / or can be used as a feature that sits on top of another application, such that snippet grouping generation can be used for content displayed in a variety of different application types. For example, these systems and methods can be built into an operating system of a computing device to allow snippet grouping generation to occur for selections made in a variety of different applications (e.g., a map application, a browser application, a social media application, etc.).

[0080] Additionally and / or alternatively, these systems and methods can be utilized by a variety of different computing devices of various types. For example, these systems and methods can be utilized by mobile computing devices, desktop computing devices, smart wearable devices (e.g., smart glasses), and / or other computing devices. These systems and methods can be used for virtual reality interfaces and augmented reality interfaces.

[0081] In some implementations, the fragment grouping may include user context data describing the context of the user when the fragment grouping was generated. For example, the computing device may include multiple sensors that can collect data about the user's context. In some implementations, physical location data of the user computing device may be obtained and stored in the fragment grouping to provide further context to the fragment. The physical location data may be provided in the graphics card and / or may be provided as an optional data set that can be viewed during fragment grouping interaction.

[0082] These systems and methods may store the segment groups locally on the user device and / or may store the segment groups on a server computing system. Local storage of the segment groups may be used to ensure that the segment groups remain private to the user and may provide offline access. Additionally, metadata associated with the collection and generation of the segment groups may remain private and secure.

[0083] These systems and methods can be provided via browser extensions, via overlay applications, and / or via built-in application features. These systems and methods can be used on mobile devices, desktop devices, smart wearable devices, and / or other computing devices.

[0084] The segment grouping may include other metadata associated with the selected portion, web page, and / or one or more contexts of the user (eg, application being used, time of day, geographic location of the user, and / or user profile data).

[0085] These systems and methods may be executed on a server computing system. Alternatively and / or in addition, these systems and methods may be executed locally on a user computing device. In some implementations, the user computing device may be communicatively connected via a network and may transmit data to perform cloud-based computing. The fragment groupings may be stored locally and / or may be stored on a server.

[0086] The systems and methods of the present disclosure provide a variety of technical effects and benefits. As an example, these systems and methods can generate and store fragment groups. Specifically, the systems and methods disclosed herein can obtain input data, determine content items (e.g., text, images, videos, and / or audio) associated with the input data, generate fragment groups, and store fragment groups. The fragment grouping may include a graphical representation of the content item, which, when selected, can guide the user to the portion of the web page from which the content item originates. The generation and storage of fragment groups can enable easy access to the stored content while maintaining links to more contexts about the content item.

[0087] Another technical benefit of the systems and methods of the present disclosure is the ability to share hierarchical information with relatively little transmission cost using fragment grouping. For example, these systems and methods can generate fragment groupings. The fragment groupings can be shared with a second user, who can initially view the content items. The second user can then select the content items to navigate to a web page and be routed to a specific portion of the web page from which the content items originate, which can allow the second user to obtain more context about the content items. Since the content items, web addresses, and text snippets can be transmitted, the provision of hierarchical information can be completed at a relatively low transmission cost. The second user can interact with the fragment groupings, view the content items individually, and then can select the fragment groupings to use the web addresses in conjunction with the text snippets to navigate to the portion of the web page with the content items highlighted or otherwise indicated. Sending the entire web page file with highlighting can include more uploads and downloads during transmission.

[0088] Another example of technical effects and benefits relates to improved computational efficiency and improved computing system functionality. For example, the systems and methods disclosed herein can utilize fragment grouping to mitigate the amount of data stored to save content items and related web page contexts. Specifically, fragment groupings can include compressed versions of content items, URLs, and text snippets, instead of saving compressed versions of complete web pages that may include a large number of content items and embedded data. In addition, searching through a collection of fragment groupings can be computationally cheaper than searching through multiple compressed web pages.

[0089] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0090] Example Apparatus and Systems

[0091] Figure 1A A block diagram of an example computing system 100 that performs segment grouping generation according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0092] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0093] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0094] In some implementations, the user computing device 102 may store or include one or more group generation models 120. For example, the group generation model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Figures 2 to 4 An example grouping generation model 120 is discussed.

[0095] In some implementations, one or more grouping generation models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single grouping generation model 120 (e.g., to perform parallel segment grouping generation across multiple instances of a segment selection).

[0096] More specifically, the grouping generation model may receive input data, determine one or more selected content items, generate segment groups, and / or determine one or more tags. In some implementations, the grouping generation model may process the selected content items and generate a summary to be added to the segment group.

[0097] Additionally or alternatively, one or more grouping generation models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the grouping generation models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a segment grouping generation service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0098] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0099] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0100] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0101] As described above, the server computing system 130 may store or otherwise include one or more machine learning packet generation models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Figures 2 to 4 Discuss example model 140 .

[0102] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0103] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0104] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques (e.g., such as error back propagation). For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0105] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0106] Specifically, the model trainer 160 may train the grouping generation model 120 and / or 140 based on a set of training data 162. The training data 162 may include, for example, training inputs (e.g., training gestures), training web pages, training text data, training image data, true value graphics cards, true value segment groups (e.g., true value segments, true value address data, and / or true value location data), and / or true value entity labels.

[0107] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some instances, this process may be referred to as personalizing the model.

[0108] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0109] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0110] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0111] In some implementations, the input of the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of image data, potential embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded representation and / or compressed representation of image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0112] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0113] In some implementations, the input of the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0114] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0115] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. The machine learning model may process the statistical data to generate an output. As an example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, the machine learning model may process the statistical data to generate a diagnostic output.

[0116] In some cases, the machine learning model may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0117] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data of one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding possibility for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted at the pixel between the images in the network input.

[0118] In some cases, the task includes encrypting or decrypting input data. In some cases, the task includes a microprocessor performing a task such as branch prediction or memory address translation.

[0119] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0120] Figure 1B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0121] Computing device 10 includes multiple applications (e.g., Application 1 to Application N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0122] like Figure 1B As shown, each application can communicate with multiple other components of the computing device (e.g., such as one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0123] Figure 1C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0124] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0125] The central intelligence layer includes multiple machine learning models. For example, Figure 1CAs shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0126] The central intelligence layer may communicate with the central device data layer. The central device data layer may be a centralized data repository for the computing device 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can use an API (e.g., a private API) to communicate with each device component.

[0127] Example Model Layout

[0128] Figure 2 An illustration of an example segment grouping generation interface 200 according to an example embodiment of the present disclosure is depicted. The segment grouping generation interface may be configured to receive a text selection, an image selection, a screenshot, a video selection, and / or an audio selection. An options menu may then pop up, the options menu having a plurality of options including a save option. The save option may be selected, which may cause a graphics card to be generated and provided for display. A segment grouping may be generated. The user may then select in which collection to save the segment grouping.

[0129] For example, at 202, a web page display window is provided for display, a portion of the web page including a text content item is selected, and a user interface element is selected to save. At 204, a segment grouping is generated, and a generated graphical representation (e.g., a graphic card with a portion of the text indicating a title and a location) is provided for display. In addition, at 204, the user interface includes a pop-up window for providing an option to add to favorites, which may include adding to a pre-existing favorite and / or generating a new favorite. These favorites may be automatically generated and / or manually generated.

[0130] At 206, a portion of the search results web page is selected (e.g., a portion of a knowledge graph may be selected), and a save option may be selected. At 208, a graphics card is provided for display, wherein the graphics card is generated based on the selected portion of the search results page. For example, the graphics card may include a picture associated with the selected portion as a background, with the selected text displayed in the foreground. Additionally and / or alternatively, the graphics card may include a search query and an associated search engine. A favorites add option may be provided.

[0131] At 210, a screenshot input is received and a save option is provided for display. At 212, a save input is received and the generated graphical representation and an option to add to favorites are provided for display. The graphical representation may include at least a portion of the screenshot and a banner including source information (e.g., a title of a web page, an entity associated with the screenshot information, and / or a source of the screenshot information).

[0132] The generated segment groupings can be stored with the generated graphical representations. The user can then select the segment groupings to view the enlarged graphical representations, view the saved content items and / or navigate to a specific point in the web page where the content items in the resource originated.

[0133] Figure 3 An illustration of an example gesture interaction 300 according to an example embodiment of the present disclosure is depicted. For example, a user input requesting a clipping interface (e.g., a selection of a "clip" user interface element 302) may be received. A clip grouping generation interface may be provided, and a graphical animation 304 may be provided to demonstrate how to select a content item. The graphical animation may demonstrate that a content item may be selected using a swipe gesture in a circular motion. An instruction user interface element 306 may be provided as a pop-up window to provide instructions on how to clip and save.

[0134] Figure 4 An illustration of an example segment grouping generation interface 400 according to an example embodiment of the present disclosure is depicted. In some implementations, a segment grouping generation interface may be requested based on received user input, and the segment grouping generation interface may be provided. The segment grouping generation interface may include clipping options or summary options that may allow segment groups to be generated based on selected content items. The segment grouping generation interface 400 may include a web page display window 402 and an interaction pane 404 that may be provided in response to web page interactions and may provide multiple web page interaction options for display. The interaction options may be predetermined and unified across web pages. Alternatively and / or in addition, the interaction options may vary based on the content of the web page, based on the selected content items, and / or based on entities associated with the web page. The interaction options may include options to add to favorites, summarize the selected portion and / or the entire web page, clip portions of the web page, save images, find more web pages similar to the current web page, compare products in the web page with other similar products, and / or track prices.

[0135] Figure 5An illustration of an example snippet grouping generation and favorites adding interface 500 according to an example embodiment of the present disclosure is depicted. The systems and methods disclosed herein may be used to generate and save snippet groupings. Snippet groupings may be added to favorites and may be searchable for later use. Snippet groupings may include generated graphics cards. In some implementations, the selected content items may be processed by one or more machine learning models to generate a summary of the content items, which may be used to generate a graphics card. The graphics card may be customizable. The graphics card may include a background color and / or background image determined based on a source web page. Alternatively and / or in addition, the background color and / or background image may be user selected. Similarly, the font may be automatically determined, predetermined, and / or user selected.

[0136] At 502, a segment grouping (including a graphical representation) is generated based on the selected portion of the web page. The generated segment grouping is then added to an "Inspo" collection of media content items and segment groups.

[0137] At 504, a summary is generated for the portion of the web page and a graphic card is generated based on the semantic understanding and / or the determined entity. The generated graphic card can be saved as part of the generated segment grouping and can be added to a favorite. The favorite can be associated with a specific entity and / or a specific type of entity.

[0138] At 506, different options are provided for sharing and / or customizing the generated segment grouping for display. For example, the template, text, and / or background of the graphical representation may be customized. Sharing options may include adding to favorites, adding to notes, sending via text message, sending via email, copying, and / or air dropping.

[0139] The snippet groupings may be posted to social media and / or posted to a localized search interface at 508. For example, a user may utilize a search application that may display a plurality of web search results in response to a query and / or may display one or more generated snippet groupings in response to a query (e.g., generated snippet groupings for a particular user and / or generated snippet groupings for associated users (e.g., friends or users near the particular user)).

[0140] The generated snippet groupings may then be shared via messaging applications, social media applications, and / or various other methods. In some implementations, the snippet groupings may be published to the web and may be used as a new format for web search results.

[0141] Fig. 9An illustration of an example favorites adding interface 900 according to an example embodiment of the present disclosure is depicted. The favorites adding interface may provide a graphical representation (e.g., a graphic card) of content items and / or segment groups for display. In response to selecting a save element, a plurality of favorites may be provided for display. The user may then add the segment grouping to a particular favorite. The segment grouping may then be stored in the particular favorite.

[0142] The particular favorite can then be opened, and a graphical representation of the generated segment grouping can be provided for display along with other graphical representations associated with other segment groupings. The favorites adding interface 900 can include displaying a graphical card (at a first size) for display when the segment grouping is generated 902. A pop-up window 904 for favorites addition can then be provided for display upon selection of the user interface element. When the segment grouping is added to the particular favorite, a favorite 906 can then be provided for display (at a second size) along with a plurality of thumbnails depicting different segment groupings in the favorite, the plurality of thumbnails including a thumbnail depicting the generated graphical card.

[0143] Fig. 10A An illustration of an example segment grouping interaction 1020 is depicted in accordance with an example embodiment of the present disclosure. Once a segment grouping is generated, the segment grouping may be interacted with to navigate to a web page associated with a content item. The location data of the segment grouping may be used to navigate to a specific portion of the web page containing the source of the content item. The content item may be highlighted when displayed. For example, a graphics card 1022 may be selected. Then, address data and location data of the segment grouping associated with the graphics card may be obtained. The address data and location data may then be utilized to open a web page 1024 to the exact location of one or more content items of the selected segment grouping, with one or more content items highlighted.

[0144] Fig. 10B An illustration of an example snippet grouping search 1040 is depicted in accordance with an example embodiment of the present disclosure. The generated snippet groupings may be provided as search results when searching locally and / or when searching the web. For example, a user may enter a search query, which may be processed to determine a plurality of suggested queries and a plurality of suggested snippet groupings, which may be provided for display (e.g., as shown in 1042) when further input may be received.

[0145] Fig. 10C Depicted is an illustration of an example highlight interface 1080 according to an example embodiment of the present disclosure. In some implementations, in response to receiving a user input requesting removal of the highlight, the highlight (eg, as shown in 1082) can be removed (eg, as shown in 1084).

[0146] Fig.11 An illustration of an example graphics card 1100 is depicted in accordance with an example embodiment of the present disclosure. The graphics card may be automatically generated and / or may be generated based on one or more user inputs. The graphics card may include a portion assigned to a content item and / or a portion assigned to an attribute of a source of the content item. The graphics card may include text, images, video, and / or audio. The graphics card may change based on the content item selected. The text may be selected text and / or may be a summary of data obtained from a web page. In some implementations, the background and / or image may be automatically selected / generated and / or may be user selected. Specifically, Fig.11 Two formats are depicted: a text-based graphic card 1102 having resource attributes and an associated title / search query; an image-based graphic card 1104 having an image, an entity thumbnail, a title or description, and resource attributes; and a screenshot-based graphic card 1106 including a screenshot, an entity logo, the title of the corresponding article, and resource attributes.

[0147] Fig.12 An illustration of an example summary snippet grouping generation interface 1200 is depicted according to an example embodiment of the present disclosure. Snippet grouping generation may include generating a graphic card. The graphic card may include text that describes a summary of selected data in a web page. The summary may be generated by processing the selected data with a machine learning model. For example, at 1202, a portion of a web page is selected. A user may select a summarize user interface element to process the selected portion to generate a summary. The summary may include language from the passage and / or may include different language that summarizes the passage in simple terms or terms that are more specific to the user. At 1204, a graphic card is generated using the summarized text to be stored with the generated snippet grouping, the graphic card including information associated with the web page and a specific location of the selected portion within the web page.

[0148] Fig.13 An illustration of an example segment grouping suggestion interface 1300 is depicted in accordance with an example embodiment of the present disclosure. In some implementations, the segment grouping suggestion interface may include providing suggested graphics cards associated with suggested segment groupings (e.g., as shown at 1304, which may be displayed in response to selection of a bookmark option as shown at 1302). The suggested segment groupings may be based on past user interactions of a particular user and / or based on past interactions of other users (e.g., popular segments). The suggested segment groupings may be selected by the user computing system and stored locally.

[0149] Fig.14An illustration of an example snippet grouping generation and sharing interface 1400 is depicted in accordance with an example embodiment of the present disclosure. A snippet grouping may be generated in response to a user selection of a content item (e.g., the selection in 1402). The generated snippet grouping may then be shared (e.g., via a text message option of a sharing interface 1404 providing multiple sharing options). The snippet grouping may be shared via a messaging application, a social media application, and / or may be inserted into another data file (e.g., a note, a text document, and / or a slideshow). The shared snippet grouping may be sent along with a graphics card and a download option for local saving (e.g., as shown in 1406). Alternatively and / or in addition, the graphics card may be selectable to navigate to one or more content items of the snippet grouping in a native web page.

[0150] Fig.15 An illustration of an example graphics card customization interface 1500 is depicted according to an example embodiment of the present disclosure. The graphics card customization interface may involve a plurality of templates that may be selected to customize a graphics card. The templates may include different images, different colors, different fonts, and / or different layouts. For example, an initial graphics card 1502 may include text in a first font, text of a first size, and a first background. A request to change the template may be received, and an enhanced graphics card 1504 may be generated with a different text font, a different text size, and / or a different background.

[0151] Fig.16 A block diagram of an example segment grouping generation system 1600 according to an example embodiment of the present disclosure is depicted. The segment grouping generation system 1600 may include obtaining input data 1602 (e.g., input data describing a selection of one or more content items). The input data 1602 may be processed to determine a selected content item 1604. Based on the determined selection, a content item may be obtained, and a graphic card 1606 may be generated. Address data 1608 may be generated and / or obtained. The address data 1608 may include uniform resource locator data. Position data 1610 may be generated and / or obtained. The position data 1610 may include text segment data, which may include a scroll position, a start of a content item, and an end of a content item, which may be used to find and highlight a content item within a source page. The graphic card 1606, the content item, the address data 1608, and the position data 1610 may be used to generate a segment grouping 1612. The segment grouping 1612 may be processed to determine one or more entity tags 1614 for the segment grouping 1612 based on the content item and / or based on the source of the content item. The entity tag 1614 may include relationship data linking the segment grouping 1612 to other segment groups associated with the same entity. The segment grouping 1612 with the entity tag 1614 may then be stored 1616 (e.g., locally and / or on a server computing system).

[0152] One or more of the determining and / or one or more of the generating may be performed based at least in part on one or more machine learning models. For example, determining the selected content item 1604, obtaining the content item and / or generating the graphics card 1606, generating the location data 1610, and / or determining the entity tag 1614 may be performed by one or more machine learning models.

[0153] Example Method

[0154] Figure 6 Depicted is a flowchart of an example method for performing according to an example embodiment of the present disclosure. Figure 6 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. The various steps of method 600 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0155] At 602, a computing system may provide data describing a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items.

[0156] At 604, the computing system may obtain input data. The input data may include a request to save one or more content items of the plurality of content items. In some implementations, the one or more content items may include at least one of an image, a video, a graphical description of a product, or audio.

[0157] At 606, the computing system may generate a segment grouping. The segment grouping may be generated based on the input data. The segment grouping may include one or more content items, address data, and location data.

[0158] One or more content items may include text data (e.g., text data describing a word, sentence, quote, and / or paragraph), image data (e.g., an image of an object (e.g., a product)), video data (e.g., a video and / or one or more frames of a video), audio data (e.g., waveform data), and / or potentially encoded data. One or more content items may include multimodal data.

[0159] In some implementations, generating a segment grouping may include obtaining one or more content items and generating a graphics card. The graphics card may describe one or more content items. The graphics card may include text data overlaid on a color and / or image. The color may be determined based on the dominant color of the web page. In some implementations, the color may be predetermined and / or may be determined based on the surrounding content items. The image may be an image determined based on the determined theme of the content item. Alternatively and / or in addition, the image may be an image close to the content item. In some implementations, the graphics card may include a font determined based on a font used in the web page. The graphics card may include a background, text describing the content item (e.g., the content item and / or a summary of the content item), and / or text and / or a logo describing the source and / or the determined entity. In some implementations, the text size of the text in the graphics card may be based on the amount of text in the content item.

[0160] The address data may describe the URL of the web page. The address data may include a uniform resource identifier and / or a uniform resource locator.

[0161] The position data may describe the position of one or more content items within a web page. In some implementations, the position data may include at least one of a scroll position, a start node, or an end node. The scroll position may describe the position of one or more content items relative to other parts of the web page. In some implementations, the start node may describe the position where one or more content items start. The end node may describe the position where one or more content items end. The position data may include a text fragment ("Scroll to Text Fragment" published by Tomayac et al. (May 20, 2022 at 9:40 p.m.), GITHUB, https: / / github.com / WICG / scroll-to-text-fragment), which may be used to indicate the position of the content item. In some implementations, the text fragment may include one or more text instructions associated with the position. The text instruction may include a string of code start data (e.g., the first text associated with the content item and / or the first pixel associated with the content item) and / or end data (e.g., the last text associated with the content item and / or the last pixel associated with the content item). The text instructions may be used to search a web page for a data set that matches the beginning and / or end of a content item.

[0162] In some implementations, the computing system may include processing one or more content items to determine entities associated with the one or more content items. An entity tag may be generated based on the entity. The segment grouping may include the entity tag.

[0163] At 608, the computing system may store the fragment grouping. The fragment grouping may be associated with a specific user. The association with the specific user may include storing the fragment grouping with metadata indicating the user and / or associating the fragment grouping with a specific user profile. The specific user may be a user who provides input data that is processed to determine the fragment grouping generation request. The fragment grouping may be stored in a user database. In some implementations, storing the fragment grouping in the user database may include storing the fragment grouping locally on the mobile computing device. Additionally and / or alternatively, a graphics card may be stored as a graphical representation of the fragment grouping. The graphics card may be automatically generated and may be customizable by the user. The graphics card may start from a template that may be customized based on other content in a web page, based on user input, and / or based on other contexts. In some implementations, the graphics card may include multimodal data. Alternatively and / or additionally, the fragment grouping may be stored on a server computing system. In some implementations, if a content item references another content item, these systems and methods may obtain additional content items and save the additional content items in the fragment grouping.

[0164] In some implementations, these systems and methods may include receiving a fragment request to provide a fragment interface. The fragment interface may include an interactive element associated with the fragment grouping. The fragment interface may be provided for display. An interface selection may be received, and the interface selection may describe a selection to select the interactive element. A portion of a web page may then be provided for display. In some implementations, the portion of the web page may include a location of one or more content items within the web page.

[0165] In some implementations, these systems and methods may include receiving an insert input. The insert input may include a user input requesting that the content item be inserted into a different interface. The segment grouping may be provided to a third party server computing system.

[0166] Additionally and / or alternatively, these systems and methods may include adding the segment grouping to favorites. The segment grouping may be added to the favorites based on received user input. Alternatively and / or additionally, the segment grouping may be automatically added to the favorites based on a determined entity associated with the content item. In some implementations, the segment grouping may be added to the favorites based on a source of the content item (e.g., a type of webpage, a type of media provider, and / or based on a type of content item).

[0167] In some implementations, segment groupings can be generated based on content items obtained from a source other than a web page, such as a mobile application, a large data file (eg, a downloaded video or book), and / or another data source.

[0168] Figure 7 Depicted is a flowchart of an example method for performing according to an example embodiment of the present disclosure. Figure 7 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. The various steps of method 700 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0169] At 702, the computing system may obtain input data. In some implementations, the input data may describe a selection of a content item associated with a segment grouping. The segment grouping may be associated with a user account of a particular user. Additionally and / or alternatively, the user account may be associated with one or more platforms. The segment grouping may include content items and deep links. Alternatively and / or additionally, the segment grouping may include segments (e.g., content items and / or media data generated based on the content items (e.g., a summary of a graphics card and / or content items)), address data (e.g., a uniform resource locator and / or uniform resource identifier), and / or metadata. The segment grouping may include location data (e.g., metadata indicating the location of the content item within a web page, a text segment for identifying start data and end data, one or more pointers, and / or one or more scroll positioning data).

[0170] In some implementations, the segment grouping may be generated by providing a graphical user interface for display. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include a plurality of content items. The segment grouping generation may include obtaining selection data. The selection data may include a request to save a content item from the plurality of content items. In some implementations, the segment grouping generation may include generating the segment grouping and storing the segment grouping in a user database.

[0171] At 704, the computing system may obtain address data and location data associated with the segment grouping. The address data may be associated with the web page. The content item may be associated with the web page. Additionally and / or alternatively, the location data may describe the location of the content item within the web page.

[0172] At 706, the computing system can obtain web page data. The web page data can be obtained based at least in part on the address data (eg, by navigating to the web page using a uniform resource locator).

[0173] The computing system can determine a location within the web page associated with the content item at 708. The location can be determined based on a text snippet, one or more pointers, and / or via web page processing.

[0174] In some implementations, the address data may include a uniform resource locator. Additionally and / or alternatively, the location data may include one or more text snippets. Then, determining a location within the web page associated with the content item may include adding the text snippet to the uniform resource locator to generate a shortcut link and inputting the shortcut link into the browser.

[0175] At 710, the computing system may provide a portion of a web page. The portion of the web page may include a location of a content item. In some implementations, providing the portion of the web page may include providing one or more indicators to the portion of the web page. The one or more indicators may indicate a content item associated with the segment grouping. In some implementations, the one or more indicators may include highlighted text related to the content item.

[0176] Figure 8 Depicted is a flowchart of an example method for performing according to an example embodiment of the present disclosure. Figure 8 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. The various steps of method 800 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0177] At 802, a computing system may provide a graphical user interface. The graphical user interface may include a graphical window for displaying a web page. In some implementations, the web page may include multiple content items.

[0178] At 804, the computing system may receive gesture data. The gesture data may describe a gesture associated with a portion of a web page. In some implementations, the gesture may include a circular gesture that encompasses the portion of the web page. The gesture data may describe a touch input to a touch screen display of a mobile computing device.

[0179] At 806, the computing system can process the gesture data to determine a selected content item. The selected content item can be associated with a portion of a web page.

[0180] In some implementations, processing the gesture data to determine the selected content item may include: determining a portion of the webpage encompassed by the gesture; determining a focus of the portion; and determining that the selected content item is associated with the focus of the portion.

[0181] Alternatively and / or in addition, processing the gesture data to determine the selected content item may include processing the gesture data with a machine learning model to determine the selected content item.

[0182] In some implementations, the selected content item may be determined based on a determined matching area associated with a rectangle determined based on a circle. The rectangle may be a rectangle of data composed based on the syntax of the web page. In some implementations, the selected content item may be determined based on determined word boundaries and / or based on determined media content boundaries. The determination may be based on hypertext markup language code boundaries. In some implementations, the source code of the web page may be parsed, and the parsed data may be processed.

[0183] Alternatively and / or in addition, the determination may include calculating the area of ​​a gesture rectangle associated with the gesture. The area of ​​one or more content item elements may be determined. The intersection area between the area of ​​the gesture rectangle and each of the areas of different content item elements may be determined. The element with the largest intersection may have the highest probability of selection and may therefore be determined to be the selected content item.

[0184] At 808, the computing system may generate a segment grouping based on the gesture data. The segment grouping may include the selected content item. In some implementations, the segment grouping may include address data and location data. The address data may be associated with a web page, and the location data may describe the location of the content item within the web page.

[0185] A save interface may then be provided for display. The save interface may provide an interactive interface element that may be selected to add the segment grouping to a favorite. A favorites interface may then be provided. A drag input may then be received that drags a graphical representation of the segment grouping to a graphical tile that depicts a particular favorite. The segment grouping may then be stored in the favorites (e.g., the segment grouping may be stored along with a relationship tag that links the segment grouping to a particular favorite). In some implementations, the graphical representation may change size and / or scale when the graphical representation of the segment grouping is dragged to a particular favorite. The size and scale changes may provide an intuitive indication of the addition to the favorite while providing an aesthetically pleasing display.

[0186] In some implementations, content items may be processed to determine entities associated with the content items. Entities may be determined by processing content items (e.g., text data, image data, audio data, video data, latent coding data, and / or link data) with a machine learning model (e.g., an image classification model, an object classification model, a text classification model (e.g., a natural language processing model), a segmentation model, a semantic model, and / or a detection model) to generate entity data (e.g., a classification). Relationship data may then be generated based on the entity data and added to the segment groupings. The relationship data may include one or more entity tags and / or references to other related segment groupings and / or related web pages or content items.

[0187] The snippet groupings may be searchable within one or more applications and / or databases. Additionally and / or alternatively, the snippet groupings may be shared via a messaging application, a social media application, and / or via another application.

[0188] In some implementations, content items may be processed with one or more machine learning models to generate tags that may be stored in the segment groups. The tags may then be used as searchable tags to display the segment groups in response to a search query. Tags may be determined based on the content of the content item (e.g., recognized words, recognized objects in an image, characteristics of a video frame or audio stream, etc.).

[0189] In some implementations, segment group generation occurs after selecting a segment group generation interface element, which can open a segment group generation interface. Alternatively and / or additionally, a user can select a content item and one or more (e.g., two) pop-ups can be provided with various options for interacting with the content item, and one of the options can include segment group generation. Alternatively and / or additionally, search results associated with the content item can be provided.

[0190] Additional public content

[0191] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0192] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents to these embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, modifications, and equivalents.

Claims

1. A computing system, the system comprising: one or more processors; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: providing data describing a graphical user interface, wherein the graphical user interface comprises a graphical window for displaying a web page, wherein the web page comprises a plurality of content items; obtaining input data, wherein the input data comprises a request to save one or more content items of the plurality of content items; Generate a fragment grouping, wherein the fragment grouping includes: the one or more content items; Address data, wherein the address data describes the URL of the web page; and location data, wherein the location data describes the location of the one or more content items within the web page; and The segment groupings are stored, wherein the segment groupings are associated with a particular user.

2. The system of claim 1, wherein the operations further comprise: receiving a fragment request to provide a fragment interface, wherein the fragment interface includes interactive elements associated with the fragment grouping; providing the fragment interface for display; receiving an interface selection selecting the interactive element; and A portion of the web page is provided for display, wherein the portion of the web page includes the location of the one or more content items within the web page.

3. The system of claim 1 or 2, wherein the operations further comprise: receiving an insert input, wherein the insert input comprises a user input requesting that the one or more content items be inserted into a different interface; as well as The segment groups are provided to a third party server computing system.

4. The system of any one of claims 1, 2, and 3, wherein generating the fragment grouping comprises: obtaining the one or more content items; generating a graphics card, wherein the graphics card describes the one or more content items; as well as The graphics card is stored as a graphical representation of the fragment grouping.

5. A system as described in one of claims 1 to 4, wherein the position data includes at least one of a scroll position, a start node, or an end node, wherein the scroll position describes the position of the one or more content items relative to other parts of the web page, wherein the start node describes the position where the one or more content items start, and wherein the end node describes the position where the one or more content items end.

6. The system of one of claims 1 to 5, wherein the one or more content items include at least one of an image, a video, a graphical description of a product, or audio.

7. The system of one of claims 1 to 6, wherein the operations further comprise: processing the one or more content items to determine entities associated with the one or more content items; generating an entity tag based on the entity; as well as The segment grouping includes the entity tag.

8. The system of one of claims 1 to 7, wherein storing the fragment grouping comprises: The segment groups are stored locally on the mobile computing device.

9. A computer-implemented method, the method comprising: obtaining, by a computing system including one or more processors, input data describing a selection of a content item associated with a segment grouping; obtaining, by the computing system, address data and position data associated with the segment grouping, wherein the address data is associated with a web page, wherein the content item is associated with the web page, and wherein the position data describes a location of the content item within the web page; obtaining, by the computing system, web page data, wherein the web page data is obtained based at least in part on the address data; determining, by the computing system, the location within the webpage associated with the content item; as well as A portion of the web page is provided by the computing system, wherein the portion of the web page includes the location of the content item.

10. The method of claim 9, wherein the address data comprises a uniform resource locator, wherein the location data comprises a text snippet, and wherein determining, by the computing system, the location within the webpage associated with the content item comprises: adding, by the computing system, the text snippet to the uniform resource locator to generate a shortcut link; as well as The computing system inputs the shortcut link into a browser.

11. The method according to claim 9 or 10, wherein the fragment grouping is generated by: providing, by the computing system, a graphical user interface, wherein the graphical user interface comprises a graphical window for displaying the web page, wherein the web page comprises a plurality of content items; obtaining, by the computing system, selection data, wherein the selection data comprises a request to save the content item from the plurality of content items; generating, by the computing system, the fragment grouping; as well as The segment groups are stored by the computing system in a user database.

12. The method of any one of claims 9, 10, and 11, wherein providing, by the computing system, the portion of the web page comprises: One or more indicators are provided to the portion of the web page, wherein the one or more indicators indicate the content items associated with the segment groupings.

13. The method of claim 12, wherein the one or more indicators include highlighted text associated with the content item.

14. The method of one of claims 9 to 13, wherein the segment grouping is associated with a user account of a specific user, wherein the user account is associated with one or more platforms.

15. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: providing a graphical user interface, wherein the graphical user interface comprises a graphical window for displaying a web page, wherein the web page comprises a plurality of content items; receiving gesture data, wherein the gesture data describes a gesture associated with a portion of the webpage; processing the gesture data to determine a selected content item, wherein the selected content item is associated with the portion of the webpage; as well as A segment grouping is generated based on the gesture data, wherein the segment grouping includes the selected content item.

16. The one or more non-transitory computer-readable media of claim 15, wherein the segment grouping includes address data and position data, wherein the address data is associated with the web page, and wherein the position data describes a location of the selected content item within the web page.

17. The one or more non-transitory computer-readable media of claim 15 or 16, wherein the gesture comprises a circular gesture surrounding the portion of the webpage.

18. The one or more non-transitory computer-readable media of one of claims 15 to 17, wherein processing the gesture data to determine the selected content item comprises: determining the portion of the webpage encompassed by the gesture; determining the focus of said section; as well as A determination is made that the selected content item is associated with the focus of the portion.

19. The one or more non-transitory computer-readable media of one of claims 15 to 18, wherein processing the gesture data to determine the selected content item comprises: The gesture data is processed with a machine learning model to determine the selected content item.

20. The one or more non-transitory computer-readable media of one of claims 15 to 19, wherein the gesture data describes a touch input to a touch screen display of a mobile computing device.

Images