Cross-application material collection method and electronic equipment

Abstract

The invention provides a cross-application material collection method and electronic equipment. The method comprises the following steps: running a first application on the electronic equipment and displaying a first interface; in response to a first operation of the user on the first interface, adding a first material in the first interface to the second application by using the first processing framework, the first operation being a three-finger glide operation in the first interface; or in response to a second operation of the user on the first interface, adding the second material in the first interface to the second application by using the second processing framework, the second operation being a long-press dragging operation on the second material selected in the first interface. In the scheme, two processing frameworks are designed, so that the purpose of cross-application collection can be achieved by calling different processing frameworks based on different user operations, the whole collection process is more flexible and diverse, various forms of collection requirements of the user can be met, and the user can conveniently collect different types of materials as required.

Description

Method and electronic device for collecting materials across applications

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on January 10, 2024, with application number 202410046020.6 and application name “Method and electronic device for collecting materials across applications”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on January 10, 2024, with application number 202410043031.9 and application name “Method and electronic device for extracting page materials”, all of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of electronic devices, and in particular to a method for collecting materials across applications and an electronic device. Background Art

[0003] Traditional electronic devices often store collections scattered across different apps, making them difficult to find. Furthermore, collections can contain a wide variety of elements, such as text, images, videos, audio, web pages, and files. To facilitate the management and use of these diverse elements across apps, developers have designed a new app that allows users to centrally manage all desired collections across apps, effectively creating a global collection experience.

[0004] However, since global collections involve different applications and different collection elements, the processing process will be cumbersome and time-consuming. Therefore, how to optimize the processing process is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides a method and electronic device for collecting materials across applications. The solution designs two processing frameworks, so that different processing frameworks can be called based on different user operations to achieve the purpose of cross-application collection, making the entire collection process more flexible and diverse, and convenient for users to collect different types of materials as needed.

[0006] In a first aspect, a method for collecting materials across applications is provided, which is applied to an electronic device, wherein the electronic device includes a first processing framework and a second processing framework. The method includes: running a first application on the electronic device and displaying a first interface; in response to a first operation of a user on the first interface, using the first processing framework to add the first material in the first interface to the second application, the first operation being a three-finger swipe down operation in the first interface; or, in response to a second operation of the user on the first interface, using the second processing framework to add the second material in the first interface to the second application, the second operation being a long press and drag operation on the second material selected in the first interface.

[0007] In the technical solution of this application, two processing frameworks are designed, so that different processing frameworks can be called based on different user operations to achieve the purpose of cross-application collection, making the entire collection process more flexible and diverse, able to meet the user's various forms of collection needs, and also convenient for users to collect different types of materials on demand.

[0008] In combination with the first aspect, in certain implementations of the first aspect, in response to a user's first operation on a first interface, when adding a first material in a first interface to a second application using a first processing framework, it may include: calling an interface of an activity management service (AMS) in the first processing framework to obtain a screenshot of the first interface; calling a global collection middleware of the first processing framework to determine a view type contained in the screenshot of the first interface; displaying a second interface on an electronic device, the second interface including a screenshot of the first interface and at least one control, the at least one control being determined based on the view type contained in the screenshot of the first interface; in response to a user clicking on a first control in at least one control, using the first processing framework to add the first material to the second application in the form of a collection corresponding to the first control. In this implementation, the cross-application collection is triggered by a three-finger swipe operation, and it is necessary to first obtain a screenshot of the interface, determine the view type therein, and then provide the user with corresponding controls to instruct the user in what form to collect the first material, and then collect the material in the form indicated by the clicked control according to the way the user clicks the control.

[0009] In combination with the first aspect, in some implementations of the first aspect, the first control is a content extraction control. In response to a user clicking on a first control in at least one control, using a first processing framework to add the first material to the second application in the form of a collection corresponding to the first control, it can include: using a global collection middleware to traverse and filter all views in the screenshot of the first interface to obtain a first subview set, the first subview set including at least one subview from which content can be extracted; using the global collection middleware, for each subview in the first subview set, content is extracted according to the view type of the subview to obtain the extracted content corresponding to the subview; when all subviews in the first subview set are extracted, a third interface is displayed on the electronic device, the third interface including the extracted content of the first subview set; in response to a user selecting a target content in the extracted content of the first subview set in the third interface, the selected target content is determined as the first material; in response to a user clicking on a confirmation collection control in the third interface, using the global collection middleware to add the first material to the second application. In this implementation, the user clicks the content extraction control, triggering the extraction of the entire screenshot's content. The process first traverses and filters all subviews in the screenshot with extractable content, then extracts the content of each subview based on its type. All extracted content is then displayed on a third interface for the user to select and add to favorites based on their selections. It should be understood that in this implementation, since it's unknown whether the user wishes to save the entire page or just a portion, or which portion they want to save, the content of all subviews in the entire screenshot is first extracted and made selectable, allowing the user to select which portion to save.

[0010] In combination with the first aspect, in certain implementations of the first aspect, when traversing and filtering all views of the screenshot of the first interface using the global collection center to obtain the first sub-view set, it can include: when traversing, for each view in the screenshot of the first interface, performing at least one of the following operations: when the view is a standard view, obtaining the view type of the view using the standard view acquisition method; or, when the view is a custom view, determining the view type of the view using the suffix of the view; and, reading the size and / or position of the view; when filtering, performing at least one of the following operations: retaining only views that are visible to the user in the screenshot of the first interface, removing hidden views in the screenshot of the first interface, and removing views that are partially or completely obscured in the screenshot of the first interface. In this implementation, some or all of the information including the type, size, and position of the view will be obtained during traversal, and when obtaining the view type, different acquisition methods will be adopted depending on whether the view is a standard view or a custom view. In this implementation, views are also screened. In addition to user-visible views, an interface may also contain hidden views or other views that are not displayed in the interface but belong to the interface. Therefore, only views visible to the user can be retained, or hidden views can be removed. The two have equivalent effects. In addition, partially or fully obscured views can also be removed.

[0011] In conjunction with the first aspect, in certain implementations of the first aspect, the first subview set includes subviews of at least one view type selected from the group consisting of text, images, offline web pages, files, videos, and links; and when using the global collection center, for each subview in the first subview set, performing content extraction based on the view type of the subview to obtain the extracted content corresponding to the subview, the process may include: for each subview, when the view type of the subview is text, calling a text extraction module in the global collection center to obtain text data of the subview, and determining the obtained text data as the subview. iew's extracted content; or, for each subview, when the view type of the subview is a picture, call the picture extraction engine in the global collection center to obtain the bitmap of the picture from the subview, and determine the bitmap as the extracted content of the subview, and / or, call the text processing module in the global collection center to obtain the text data contained in the picture by optical character recognition (OCR); or, for each subview, when the view type of the subview is an offline web page, call the web page extraction engine in the global collection center to extract the title and Content parsing to obtain the extracted content of the subview, the extracted content of the subview includes the extracted title and the parsed web page content; or, for each subview, when the view type of the subview is file, call the file extraction engine in the global collection center to obtain the file path of the file, and determine the copy of the file or the file path of the file as the extracted content of the subview, and determine the file information of the file as the extracted content of the subview; or, for each subview, when the view type of the subview is video, call the video extraction engine in the global collection center The engine reads the video identifier (ID) written into the barrier-free node by the first application, identifies the playback area of ​​the video, and records the screen, takes screenshots and / or adds tags to obtain the extracted content of the subview; or, for each subview, when the view type of the subview is a link, calls the online link extraction engine in the global collection center, obtains the link information according to the uniform resource locator (URL) field carried in the activity or the web view webview's read uniform resource locator (getURL) method, and the obtained link information is the extracted content of the subview. In this implementation method, there will be different content extraction strategies for different types of views, so that the present application solution can simultaneously support content extraction of multiple views, rather than just content extraction of a single view.

[0012] In combination with the first aspect, in certain implementations of the first aspect, for each subview, when the view type of the subview is text, calling the text extraction module in the global collection center to obtain the text data of the subview, and determining the obtained text data as the extracted content of the subview, can include: using the text extraction module to call the intelligent recognition platform, so that the intelligent recognition platform obtains the text data of the subview through optical character recognition (OCR), or, using the text extraction module, obtaining the text data of the subview by reading the information written to the accessibility node by the first application. In other words, for text-type subviews, there are two ways to extract content: OCR recognition or reading from the accessibility node.

[0013] In combination with the first aspect, in certain implementations of the first aspect, in response to a user clicking on the confirmation collection control in the third interface, using the global collection middle platform to add the first material to the second application, at least one of the following operations may be included: directly adding the text material in the first material to the second application; storing the image material in the first material to the gallery application, and adding the thumbnail of the image to the second application, the image stored in the gallery application is associated with the thumbnail of the image in the second application, and the thumbnail of the image is the mapping element of the image in the second application; storing the offline web page material in the first material to the note application, and adding the introduction content of the offline web page to the second application, and the offline web page stored in the note application is associated with the introduction content of the offline web page in the second application , the introduction content of the offline webpage is the mapping element of the offline webpage in the second application; the video material in the first material is stored in the gallery application, and the thumbnail of the first frame image of the video is added to the second application, the video stored in the gallery application is associated with the thumbnail of the first frame image of the video in the second application, and the thumbnail of the first frame image of the video is the mapping element of the video in the second application; the file material in the first material is stored in the file management application, and the file information of the file is added to the second application, the file stored in the second application is associated with the file information of the file in the second application, and the file information of the file is the mapping element of the file in the second application, and the file information includes the icon and file name of the file type; and, the link material in the first material is directly added to the second application. In this implementation method, instructions are given for completing the collection of materials of different view types (different types of materials) in the first material according to different adding schemes.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the first control is a web page collection control, a link collection control, or a file collection control. In response to a user click operation on the first control of at least one control, using the first processing framework to add the first material to the second application in the form of a collection corresponding to the first control can include: when the first control is a web page collection control, adding the first material to the second application in the form of an offline web page, and the first material is the content of the entire web page displayed in the first interface; or, when the first control is a web page collection control, adding the first material to the second application in the form of an offline web page, and the first material is the content displayed on the interface of the web page displayed in the first interface; or, when the first control is a link collection control, adding the first material to the second application in the form of a link, and the first material is an online link of the web page displayed in the first interface; or, when the first control is a file collection control, adding the first material to the second application in the form of a file, and the first material is the file displayed in the first interface, and the file is an open document, a full-screen video, or a full-screen picture. In this implementation, examples of other collection forms are given, which enriches the form of collection content and makes it easier for users to collect according to different needs. In other words, users can collect the content they want to collect in different ways according to different collection needs.

[0015] In conjunction with the first aspect, in certain implementations of the first aspect, when calling the global collection middle platform of the first processing framework to determine the view type contained in the screenshot of the first interface, it can include: determining the view type contained in the screenshot of the first interface according to the whitelist configuration of the first application or the whitelist configuration of the first interface. In this implementation, the view type contained in the screenshot is determined according to the whitelist configuration. The introduction of the whitelist configuration can help determine the extractable range in the interface, so that the extracted content adds customized constraints, and different whitelists can be set for different applications or different interfaces as needed.

[0016] In combination with the first aspect, in certain implementations of the first aspect, when a second interface is displayed on an electronic device and the second interface includes a screenshot of the first interface and at least one control, it may include: when the view type included in the screenshot of the first interface includes a web page, at least one control includes at least one of a web page collection control, a link collection control, and a content extraction control; or, when the view type included in the screenshot of the first interface includes a file, at least one control includes at least one of a file collection control and a content extraction control; or, when the view type included in the screenshot of the first interface includes only pictures or videos, at least one control includes at least one of a file collection control and a content extraction control. In this implementation, the type of control displayed on the second interface will be determined based on the different view types included in the screenshot, so that the user can select as needed. For example, assuming that the first interface does not include a web page view, there is no need to display a web page collection control on the second interface.

[0017] In combination with the first aspect, in certain implementations of the first aspect, in response to the second operation of the user on the first interface, using the second processing framework to add the second material in the first interface to the second application, it can include: using the content recognition service in the second processing framework to obtain the second material selected by the user in the first interface; using the content recognition service to determine the view type contained in the second material based on the long press coordinates of the second operation; using the view identifier in the second processing framework to traverse the view tree to obtain the subviews of the extractable content contained in the second material, and obtain a second subview set, which includes at least one subview of the extractable content; using the content recognition service to extract content from each subview in the second subview set according to the view type of the subview, and obtain the extracted content corresponding to the subview; when all subviews in the second subview set are extracted, the extracted content of the second subview set is obtained; using the window management service (WMS) in the second processing framework to add the extracted content of the second subview set to the second application based on the view type contained in the extracted content of the second subview set. In this implementation, a global collection is triggered by a drag operation. Because dragging requires selecting an object before dragging, and the coordinates of the drag's starting point (the coordinates of a long press) are available, the selected subview is determined by traversing the view tree to find all subviews, rather than identifying all subviews in the entire interface. This implementation allows the user to first select the object they want to collect (the second material), then performs view identification on the second material, extracts the content within the view, and then adds the extracted content to the collection.

[0018] In combination with the first aspect, in certain implementations of the first aspect, when using the window management service in the second processing framework, according to the view type contained in the extracted content of the second subview set, the extracted content of the second subview set is added to the second application, it can include at least one of the following operations: directly adding the text content in the extracted content of the second subview set to the second application; storing the image content in the extracted content of the second subview set to the gallery application, and adding the thumbnail of the image to the second application, the image stored in the gallery application is associated with the thumbnail of the image in the second application, and the thumbnail of the image is the mapping element of the image in the second application; storing the offline web page content in the extracted content of the second subview set to the note application, and adding the introduction content of the offline web page to the second application, the offline web page stored in the note application is associated with the thumbnail of the image in the second application The introduction content of the offline web page is associated, and the introduction content of the offline web page is the mapping element of the offline web page in the second application; the video content in the extracted content of the second subview set is stored in the gallery application, and the thumbnail of the first frame image of the video is added to the second application, the video stored in the gallery application is associated with the thumbnail of the first frame image of the video in the second application, and the thumbnail of the first frame image of the video is the mapping element of the video in the second application; the file content in the extracted content of the second subview set is stored in the file management application, and the file information of the file is added to the second application, the file stored in the second application is associated with the file information of the file in the second application, the file information of the file is the mapping element of the file in the second application, and the file information includes the file type icon and file name; and the link content in the extracted content of the second subview set is directly added to the second application.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, after adding the first or second material to the second application, the method further includes: displaying a snackbar on the electronic device, the snackbar including a prompt indicating successful addition and a viewing control; and in response to a user clicking the viewing control, launching the second application and displaying the addition result in the interface of the second application. This implementation provides the execution process of prompting the user of the successful addition and facilitating the user to view the addition result after the addition is successful.

[0020] In conjunction with the first aspect, in certain implementations of the first aspect, the first material is obtained by executing the following steps: obtaining a first object of a first interface, wherein the first object contains a view view of the material to be extracted; determining a material type set of the first object, wherein the material type set includes at least one material type selected from text, images, files, web pages, videos, and online links; determining a view set corresponding to each material in the material type set in the first object, wherein each material corresponds to a view set, and each view set includes all views from which materials of the material type corresponding to the view set can be extracted; for each material type in the material type set of the first object, extracting materials of the material type from the view set corresponding to the material type; when all view sets corresponding to the material type set of the first object are extracted, obtaining the first material corresponding to the first object. In this implementation, views that include the same material type are first aggregated according to material type, and then, for each material type, the material of the material type is extracted from all views that include the material type. Since the extraction methods for the same material type are the same or similar or the modules used for execution are the same, this implementation can improve extraction efficiency.

[0021] In a second aspect, a method for extracting page materials is provided, the method comprising: obtaining a first object from a first page, the first page being a display interface of a first application running on an electronic device, the first object containing a view of the material to be extracted; determining a material type set of the first object, the material type set including at least one material type of text, picture, file, web page, video and online link; determining a view set corresponding to each material in the material type set in the first object, each material corresponding to a view set, each view set including all views from which materials of the material type corresponding to the view set can be extracted; for each material type in the material type set of the first object, extracting materials of the material type from the view set corresponding to the material type; when all view sets corresponding to the material type set of the first object are extracted, the first material corresponding to the first object is obtained.

[0022] In this technical solution, the selected object's view containing the material type is identified based on the selected object's material type, and then the material is extracted by classification, meeting diverse material extraction needs. The extracted page material can then be used to collect, merge documents, or share with other applications or other devices.

[0023] It should be noted that the extraction method of the second aspect can be used to extract the first material of the first aspect according to the material type during the execution of the method of the first aspect. The first page of the second aspect can be the first interface of the first aspect, and the first object of the second aspect can be a screenshot of the first interface of the first aspect. The first material of the second aspect can be the first material of the first aspect (corresponding to the case of a three-finger swipe down operation).

[0024] It should also be understood that the extraction method of the second aspect can also be used to extract corresponding content from the second material according to the material type during the execution of the method of the first aspect. The extraction process is similar to the extraction process of the first material. In this case, it is only necessary to replace the "first object" in the scheme for obtaining the first material with the "second material", and to replace the "first material corresponding to the first object" with the "extracted content of the second sub-view set". For the sake of brevity, it will not be repeated.

[0025] In combination with the second aspect, in certain implementations of the second aspect, when the material type set of the first object includes text material, the view set corresponding to the text material includes at least one of text view, image view, file view and webview; when extracting the material of each material type in the material type set of the first object from the view set corresponding to the material type, it may include: when extracting the text material from the view set corresponding to the text material, based on the acquisition method of the first object, extracting the text data in each view in the view set, and determining the extracted text data as the text material extracted from the first object. In this implementation, for the text material, the text view, image view, file view and webview containing text data will be found from the first object and added to its view set. When extracting, the extraction will be based on the acquisition method of the first object, because the different acquisition methods of the first object will play a certain role in determining the content contained in the view set.

[0026] In conjunction with the second aspect, in certain implementations of the second aspect, when extracting text material from a view set corresponding to the text material, extracting text data from each view in the view set based on the acquisition method of the first object, and determining the extracted text data as the extracted text material may include:

[0027] When the first object is obtained by a long-press and drag operation on the first object, the view set corresponding to the text material includes a text view and also includes a file view and / or a webview, and text data in the text view is extracted according to a preconfigured extraction rule in a configuration file corresponding to the first object, or, when there is no corresponding configuration file for the first object, text data in the text view is extracted according to a custom extraction rule; or,

[0028] When the first object is obtained by a three-finger swipe down operation in the interface, the view set corresponding to the text material includes an image view of the interface screenshot of the first page, and text is extracted from the image view by optical character recognition (OCR).

[0029] In this implementation, for a long-press drag operation, since the coordinates of the long-press can be determined, and the case where text can be dragged out only includes dragging after selecting a text paragraph in a normal interface, file, or webpage display interface, when a long-press drag operation is performed, the view collection must include the text view and may also include the file view or webview corresponding to the first interface. In this case, the text data in the text view can be extracted according to preconfigured or customized extraction rules. It should be understood that when a text paragraph is selected in a normal interface, an open file, or an open webpage, it is equivalent to the file view and webview containing the text view. Since the first object is dragged after it has been selected, it will only be a portion of the first interface, not the entire interface. Therefore, when extracting text material in a long-press drag operation, it is not necessary to consider the view corresponding to the entire interface, but only the text view within the selected content. For a three-finger swipe operation, since it is an operation on the entire page and no content is pre-selected, the first object may correspond to the entire page, or the entire webpage or file corresponding to the page. Therefore, in a three-finger swipe operation, the text material corresponds to the entire image view of the interface screenshot.

[0030] In conjunction with the second aspect, in certain implementations of the second aspect, when the first object does not have a corresponding configuration file, extracting text data from the text view according to a custom extraction rule may include:

[0031] Read the text data of the text view from the accessibility node (AccessibilityNodeInfo) corresponding to the text view; or,

[0032] When the text data of the text view cannot be read from AccessibilityNodeInfo, the text content is read from the text acquisition interface (getText) corresponding to the text view; or,

[0033] When the text data of the text view cannot be read from AccessibilityNodeInfo and getText, the text view identifier (ViewID) is obtained by directly calling view, and then the text data of the text view is obtained according to the text ViewID; or

[0034] When the text data cannot be read from AccessbilityNodeInfo, getText and ViewID, it is obtained from the view character acquisition interface (getViewString) of the text view by directly calling the view.

[0035] In this implementation, custom extraction rules are some execution strategies sorted by execution priority. First, read from the barrier-free node. This requires relying on the first application to write data to the barrier-free node. Most current applications meet this condition. When encountering a few applications that do not meet this condition or fail to read for other reasons, other ways of reading data are tried in turn. The above strategies are from simple to complex from front to back, and the simplest reading method is prioritized.

[0036] In combination with the second aspect, in certain implementations of the second aspect, when the material type set of the first object includes picture materials, the view set corresponding to the picture materials includes at least one of picture view, webview and custom view; for each material type in the material type set of the first object, when extracting the material of the material type from the view set corresponding to the material type, it can include: according to the configuration corresponding to the view name or application name, determining at least one view containing the picture from the view set corresponding to the picture material, including the standard picture view, webview and custom view; for the standard picture view, obtaining the bitmap of the standard picture view by reading the storage interface of the picture, and the bitmap of the standard picture view and the uniform resource identifier URI of the bitmap of the standard picture view. Determine the image material extracted from the first object; for the webview, obtain the bitmap and URI of the image in the webview through JS script injection, and determine the bitmap of the webview and the URI of the bitmap of the webview as the image material extracted from the first object; for the custom view, copy the bitmap by traversing and searching for the key attributes in the custom view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the image material extracted from the first object; for the custom image view, when the bitmap cannot be found from the key attributes, draw the bitmap of the custom image view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the image material extracted from the first object.

[0037] In this implementation, different extraction strategies are determined and executed based on whether the view is an image view or a standard image view. A standard image view is an image view, and images may also exist in a web view, so they are explained separately. For non-standard image views, that is, custom views, if the bitmap can be found, its bitmap is read; if not, the bitmap of the custom view is directly drawn. Figure 7 shows an example of the image extraction process. The above extraction strategy can extract image materials regardless of whether it is a standard image view, and by using the strategy of directly drawing the bitmap as a backup, it can ensure that images in all views containing images can be extracted.

[0038] In combination with the second aspect, in certain implementations of the second aspect, for a standard image view, when obtaining the bitmap of the standard image view by reading the storage interface of the image, it can include: reading the bitmap of the standard image view from the storage node of the standard image view; or, when the bitmap of the standard image view cannot be read, drawing the bitmap of the standard image view.

[0039] This implementation provides a strategy for searching for bitmaps of standard images, prioritizing reading from storage nodes. If the bitmap cannot be found, the bitmap is directly drawn. This strategy provides a fallback for directly drawing the bitmap even if the bitmap cannot be read, ensuring that images can be extracted from all views containing them.

[0040] In combination with the second aspect, in some implementations of the second aspect, when the material type set of the first object includes file material, the view set corresponding to the file material includes file view; for each material type in the material type set of the first object, when the material of the material type is extracted from the view set corresponding to the material type, it can include: when the first object is a file selected in the first page, the acquisition method of the first object is a long press and drag operation on the first object, and the view set corresponding to the file material only includes file view, directly extract the file information of the file view in the first object, and find out whether the file exists in the storage path of the file view; when the file exists in the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object; or, when the first page is a display page for opening a file, the first object For the open file, the first object is obtained by a three-finger swipe down operation on the first page. When the view set corresponding to the file material includes the file view corresponding to the open file, and also includes a text view and / or a picture view, the file information of the open file is searched from the pre-storage space, and the file is copied according to the file path indicated by the read file information, and the URI of the copied file and the read file information are determined as the file material extracted from the first object, or, when the file information of the open file cannot be found in the pre-storage space within a preset time period after the three-finger swipe down operation occurs, the file information of the open file is obtained through the field of the activity activity, and the file is copied according to the file path indicated by the obtained file information, and the URI of the copied file and the obtained file information are determined as the file material extracted from the first object; the pre-storage space includes file information of all detected open files.

[0041] In this implementation, different file view extraction strategies are provided for the dragging scenario and the three-finger swipe scenario. For file collection, in the dragging scenario, one or more files contained in the interface must be selected and dragged. Whether these files are open or not does not matter, so only file information extraction and other operations are required. For the three-finger swipe scenario, an interface (the first page) must be currently displayed, in which files are being opened and displayed. The three-finger swipe is a three-finger swipe on the opened file. At this time, the file view in the first selected object must be the file view corresponding to the opened and displayed file. At this time, it is necessary to find the file information of this file from the pre-stored file information of all open files, and then trace back to this file to copy and extract the URI.

[0042] In combination with the second aspect, in certain implementations of the second aspect, when the first object is a file selected in the first page, the method for obtaining the first object is a long press and drag operation on the first object, and the view set corresponding to the file material only includes the file view, the file information of the file view in the first object is directly extracted, and whether the file exists under the storage path of the file view is searched; when the file exists under the storage path of the file view, the file is copied, and the URI of the copied file and the extracted file information are determined as the file material extracted from the first object, which may include: determining whether the first application is connected to the file drag framework; when the first application is connected to the file drag framework, directly obtaining the URI of the open file; or, when the first application is not connected to the file drag framework, scanning the file path of the open file in the electronic device through the file name of the open file, thereby obtaining the URI of the open file.

[0043] In this implementation, if the application (first application) itself supports dragging and dropping files, the URI of the file can be found directly, but if the application does not support dragging and dropping files, it is necessary to traverse the file system to find the file.

[0044] In combination with the second aspect, in certain implementations of the second aspect, when the material type set of the first object includes web page material, the view set corresponding to the web page material includes webview; for each material type in the material type set of the first object, when extracting the material of the material type from the view set corresponding to the material type, it can include: extracting the title and text content of the webview in the first object by reading and parsing a readable readability file, and determining the extracted title and text content as the web page material extracted from the first object.

[0045] In this implementation, for webview, web page materials in the form of offline web pages are extracted by parsing the content therein.

[0046] In combination with the second aspect, in certain implementations of the second aspect, when the material type set of the first object includes video material, the view set corresponding to the video material includes webview; for each material type in the material type set of the first object, when extracting the material of the material type from the view set corresponding to the material type, it can include: determining the video playback area in the first page by reading the video configuration information written by the first application in the accessibility node, and performing at least one of recording, screenshoting and adding tags to the video being played to obtain video excerpt content, and determining the obtained video excerpt content and the URI of the video excerpt content as the video material extracted from the first object.

[0047] In this implementation, the video material must be a partial or complete excerpt of the video played in weview or something similar to webview. The excerpt is a process, so it is equivalent to recording and extracting for a period of time. First, lock the video playback area (for example, when the video is not played in full screen on the web page), and then start recording the screen, taking screenshots, etc. to extract a section or all of the video, and you will get the video excerpt material, which can also be called video material.

[0048] It should be noted that in addition to storing video excerpts, you can also store online links to the video. In this case, the online link material is extracted according to the online link extraction method, which does not belong to the extraction of video material. In addition, when transferring video files between chat applications, the video exists in file form. You need to select the video file and then drag it according to the drag and drop scenario. Therefore, the same method can be used to extract file materials, except that the extracted file material is a video.

[0049] In combination with the second aspect, in certain implementations of the second aspect, when the material type set of the first object includes online link materials, the view set corresponding to the online link materials includes at least one of an online link view and a webview; for each material type in the material type set of the first object, the material of the material type is extracted from the view set corresponding to the material type, including: obtaining the online link material according to the uniform resource locator (URL) field carried in the activity module or reading the uniform resource locator (getURL) from the webview.

[0050] In this implementation, online links can be found in the activity's current field or read from the webview. Some views are not webviews but have similar properties and corresponding links, such as the online editing of shared documents that often occurs in work applications. Other scenarios are not listed here.

[0051] In one example, when obtaining online link materials according to the URL field carried in the activity or the getURL method from the webview, it can include: when the webview corresponding to the first page is a standard webview, directly using the getURL method to obtain the online link of the standard webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is a custom webview, matching the name of the webview from the configuration information of the first page, finding the custom webview corresponding to the first page, and then using the getURL method to obtain the online link of the custom webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is neither a standard webview nor a custom webview, obtaining the online link of the webview from the URL field carried in the activity.

[0052] In this example, different methods of obtaining online links are adopted depending on whether the text view is a standard webview, a custom webview, or neither.

[0053] In a third aspect, a device for collecting materials across applications is provided, the device comprising a unit composed of software and / or hardware for executing any one of the methods of the first aspect or the second aspect.

[0054] In a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the methods of the first aspect or the second aspect can be implemented.

[0055] In a fifth aspect, a chip is provided, comprising a processor, wherein the processor is configured to read and execute a computer program stored in a memory, and wherein when the computer program is executed by the processor, any one of the methods of the first aspect or the second aspect can be implemented.

[0056] Optionally, the chip further includes a memory, and the memory is electrically connected to the processor.

[0057] Optionally, the chip may further include a communication interface.

[0058] In a sixth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it can implement any one of the methods of the first aspect or the second aspect.

[0059] In a seventh aspect, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it can implement any one of the methods of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] FIG1 is a schematic diagram of an interactive process of collecting materials across applications according to an embodiment of the present application.

[0061] FIG2 and FIG3 are schematic diagrams of another interactive process of collecting materials across applications according to an embodiment of the present application.

[0062] FIG4 is a schematic diagram of another interactive process of collecting materials across applications according to an embodiment of the present application.

[0063] FIG5 is a schematic diagram of another interactive process of collecting materials across applications according to an embodiment of the present application.

[0064] FIG6 is a schematic diagram of a processing framework for collecting cross-application materials according to an embodiment of the present application.

[0065] FIG7 is a schematic diagram of another processing framework for cross-application collection materials according to an embodiment of the present application.

[0066] FIG8 is a schematic flowchart of a method for collecting materials across applications according to an embodiment of the present application.

[0067] FIG9 is a schematic flow chart of a method for extracting page materials according to an embodiment of the present application.

[0068] FIG10 is a schematic diagram of a page material extraction framework in a long-press and drag operation scenario according to an embodiment of the present application.

[0069] FIG11 is a schematic diagram of an execution framework for extracting text materials based on OCR according to an embodiment of the present application.

[0070] FIG12 is a schematic flowchart of extracting image material according to an embodiment of the present application.

[0071] FIG13 is a schematic diagram of an execution framework for extracting file materials in a three-finger swipe scenario according to an embodiment of the present application.

[0072] FIG14 is a schematic flowchart of extracting file materials in a three-finger swipe scenario according to an embodiment of the present application.

[0073] FIG15 is a schematic diagram of an execution framework for extracting web page materials according to an embodiment of the present application.

[0074] FIG16 is a schematic diagram of an execution framework for extracting online links according to an embodiment of the present application.

[0075] FIG17 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0076] FIG18 is a schematic diagram of the hardware structure of another electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following describes the solutions of the embodiments of the present application with reference to the accompanying drawings.

[0078] FIG1 is a schematic diagram of an interactive process for collecting materials across applications according to an embodiment of the present application. As shown in FIG1 , assume that a user opens application A on an electronic device and enters the interface shown in interface 101, which includes some graphic data of a news item. Assume that the user selects a portion of content in interface 101 by pressing and sliding, as shown in interface 102, a portion of text content and an image content are selected. Assume that the user raises their hand after selecting the content and then long-presses the selected content area. In response to the long-press operation, the electronic device displays interface 103, in which the selected content is superimposed in the interface in a floating form. Assume that the user drags the floating selected content to the side of the electronic device after long-pressing in interface 103. The selected content can be added to the collection space application. As shown in interface 104, the display interface of the collection space application is superimposed on the interface of application A. Here, it is a long strip interface. From the display interface of the collection space, it can be seen that the selected content has been added to the collection space, and the text and image are stored as a material respectively.

[0079] It should also be understood that Figure 1 uses application A as an example, where the page content in application A is added to the favorites space for collection. For other applications, page content can also be added to the favorites space, and this will not be repeated. For example, you can select and drag chat content in a chat application to add it to the favorites space. Another example is that you can drag and drop files transferred between chat applications to the favorites space. Another example is that you can drag and drop attachments in an email application to the favorites space. Another example is that you can select and drag the body content of an email in an email application to the favorites space. Another example is that you can drag and drop an image clicked in a shopping application to the favorites space. Another example is that you can drag and drop images and videos in a gallery application to the favorites space. Other examples are not listed one by one.

[0080] Figures 2 and 3 are schematic diagrams of another interactive process of collecting materials across applications in an embodiment of the present application. As shown in Figure 2, it is assumed that the user opens application A on an electronic device and enters the interface shown in interface 104, which includes some graphic data of a news item. Assume that the user performs a three-finger long press and then a three-finger swipe operation in interface 104. In response to the three-finger swipe operation, the electronic device displays interface 105. In addition to the interface content of interface 104, interface 105 also includes three controls: web page collection, content extraction, and screenshot collection. It should be understood that in practice, the number, type, and form of controls included in interface 105 can be set as needed. It should also be understood that the number and type of controls in interface 105 may also be different depending on the content contained in interface 104. For example, here interface 104 is a web page, so interface 105 includes web page collections and may also include link collections. Assuming that interface 104 displays an open file, for example, a PDF document, interface 105 may include at least one of file collection, content extraction, and screenshot collection. Furthermore, assuming that interface 104 displays a full-screen image, interface 105 may include image collection (equivalent to file collection) and content extraction (here, simply referring to the extraction of content from the image). Other scenarios are not listed one by one.

[0081] Suppose a user clicks the content extraction control in interface 105. In response to this click, the electronic device displays interface 106. Throughout interface 106, the background color of selectable content differs from the background color of unselectable content. Here, the background color of unselectable content is gray, while the background color of selectable content is white. Interface 105 also includes two controls: Exit and Save. Suppose a user selects a portion of the selectable content in interface 105 by pressing and sliding, as shown in interface 107. As can be seen from interface 107, a text segment (text content #1) and two images (image content #1 and image content #2) are selected as examples. The background color of the selected content is different from the background color of unselected content and the background color of unselectable content. In other words, the selected content is marked with a background color in interface 107. However, it should be understood that other methods can be used to mark the selected content in practice. For example, the color of the selected target content can be different from the color of the other selectable content and also different from the color of the other unselectable content.

[0082] As shown in Figure 3, it is assumed that the user raises his hand to end the selection operation after pressing and sliding in interface 107, and clicks the collection control. In response to the click operation, the electronic device adds the selected content to the collection space. When the addition is completed, interface 108 is displayed. Interface 108 includes a floating capsule (snackbar) #1. The floating capsule #1 includes a prompt message indicating successful collection and a click-to-view control indicating that the collection results can be viewed. It is assumed that the user clicks the view control "Click to View" in interface 108. In response to the click operation, interface 109 is displayed on the electronic device. As shown in interface 109, the display interface of the collection space application is superimposed on the interface of application A. Here, the collection space application is a long strip interface, and it can be seen from the display interface of the collection space that the selected content has been added to the collection space, and the text and picture are stored as one material respectively. The selected text and two pictures shown in interface 107 are both collected in the collection space.

[0083] It should also be understood that Figures 2 and 3 take application A as an example. After being triggered by swiping down with three fingers, the page content in application A is added to the collection space for collection. For other applications, the page content therein can also be added to the collection space, which will not be repeated here. For example, you can swipe down with three fingers on the chat interface in a chat application, and after selecting content extraction, select the target content, and then collect the target content into the collection space by clicking the collection control. For another example, you can swipe down with three fingers on the display interface of an email application, and then select materials to add to the collection space. The interaction process is similar to Figures 2 and 3, which will not be repeated here. For another example, you can swipe down with three fingers in a shopping application, and then select pictures and other materials to add to the collection space. For another example, you can swipe down with three fingers on the display interface of a gallery application to add all pictures and / or videos in the current interface to the collection space. Other situations will not be listed one by one.

[0084] Assuming the user clicks on the webpage collection space on interface 105, the entire webpage corresponding to interface 104 can be saved to the collection space as an offline webpage material. Assuming the user clicks on the screenshot collection on interface 105, the user can take a screenshot of interface 104 and save it to the collection space as an image material. Assuming interface 105 also includes a link collection control, when the user clicks on the link collection control, the online link of the webpage corresponding to interface 104 can be saved to the collection space as an online link material. Other situations are not listed one by one.

[0085] FIG4 is a schematic diagram of another interactive process of collecting materials across applications in an embodiment of the present application. FIG4 can be seen as an interactive operation performed after the interface 105 shown in FIG2 is displayed. Assuming that the user clicks on the web page collection in the interface 105, in response to the click operation, the electronic device displays the interface 110, and the interface 110 includes the floating capsule #2. Assuming that the user clicks on "Click to View" in the floating capsule #2, in response to the click operation, the electronic device displays the interface 111. It can be seen in the interface 111 that the materials added to the collection space are collected in the form of web pages, that is, offline web page materials. By comparing the interface 111 and the interface 109, it can be seen that when the web page is collected, the valid content displayed in the entire interface 101, or the valid content displayed in the entire interface 105, is added to the collection space, and is stored as a group of materials in the collection space, rather than storing text and pictures separately.

[0086] It should also be understood that webpage collection refers to the collection of the content displayed on interface 105 as a webpage, rather than the collection of all the contents of the entire webpage involved in interface 105. Alternatively, webpage collection refers to the collection of all the contents of the entire webpage involved in interface 105 as a webpage, rather than being limited to the content visible to the user on the screen of interface 105. The above two methods can be pre-set according to the actual needs of the electronic device.

[0087] In addition, assuming that the user clicks on the screenshot collection in the interface 105, the screenshot obtained after the screenshot of the interface 105 can also be collected into the collection space.

[0088] Figure 5 is a schematic diagram of another interactive process of collecting materials across applications in an embodiment of the present application. Figure 5 can be seen as an interactive process performed after the interface 103 shown in Figure 1 is displayed. Assuming that the user presses and holds the floating selected content in the interface 103 and drags it to the side of the electronic device, the electronic device follows the hand to display the door opening effect shown in the interface 112, which includes the icon of the collection space application in the portal application. Assuming that the user continues to drag to the collection space icon and then raises his hand, the electronic device responds to the operation to execute the step of adding the selected content to the collection space, and displays the interface 113 after adding it. In the interface 113, the user is prompted that the collection has been successful through the floating capsule #3. Assume that the user clicks "Click to view" in floating capsule #3. In response to the click operation, interface 114 is displayed on the electronic device. As shown in interface 114, the display interface of the collection space application is superimposed on the interface of application A. Here, the collection space application is a long strip interface, and from the display interface of the collection space, it can be seen that the selected content is added to the collection space, and the text and picture are stored as a material respectively. The selected text and picture shown in interface 103 are both collected in the collection space, with the text as one material and the picture as another material.

[0089] It should be understood that Figures 1 to 5 are only examples of the interactive process of cross-application collection. In practice, the specific content of the interface and the controls included and the number of controls may vary. As can be seen from Figures 1 to 5, the process of cross-application collection of materials can collect different contents of different applications into the collection space, that is, to achieve global collection, which is convenient for users to view and use the collected materials in the collection space later. In the above-mentioned interactive process, different materials will be added to the collection space in different ways under different trigger operations, so the electronic device needs to have the ability to extract the content of different types of materials. In response to the above problems, the cross-application collection material solution provided in this application will extract content according to different types of views, thereby realizing the content extraction of the above-mentioned different materials.

[0090] Figure 6 is a schematic diagram of a processing framework for collecting assets across applications, according to an embodiment of the present application. As shown in Figure 6, the InputManager is used to capture three-finger presses (used to launch the global collection service) and three-finger swipes (used to trigger the global collection service). The HwFingersSnapshooter module in the InputProcessor is used to identify these gestures.

[0091] The input processor sends the various operations obtained to the global collection service (global collection middle platform), which includes the collection service (CollectionService) module, the sensor manager (SensorManager), the gesture (gesture) module and various view extraction engines / modules. For example, it includes the text extraction module (TexExtractClient), the picture extraction engine (PictureExtractEngine), the file extraction engine (FileExtractEngine), the video extraction engine (VideoExtractEngine), the online link extraction engine (OnlinelinkExtractEngine) and the web page extraction engine (WebPageExtractEngine). In other words, each view corresponds to an extraction engine / extraction module. Therefore, the number of extraction engines / extraction modules included in the global collection middle platform can be set according to the different types of views that need to be extracted.

[0092] Each extraction engine / module in the global collection center extracts views under the trigger of the gesture module, and starts and schedules the corresponding processes through the activity manager service (AMS) to perform subsequent operations.

[0093] The global collection center can request a node (resquestContentNode) from AMS. AMS passes the activity instance based on the request feedback node, thereby sending a screenshot of the interface during the three-finger swipe down operation to the global collection center.

[0094] After extracting the content to be collected (which may be an entire web page, file, etc., or a small part of the content in the first interface), the global collection center stores different types of materials in the file manager, gallery, notes or collection space according to the type of view, and stores mapping elements such as thumbnails in the collection space. The collection space application can display, organize and search, preview / share the collection content, and can share the collection content with other electronic devices through multi-device sharing. When it is necessary to extract text from the image, the global collection platform sends the initial data, that is, the data to be recognized, to the intelligent recognition platform AIPlugin, and receives the recognition results. The intelligent recognition platform includes vision service (VisionService), text extraction (TextConfiguration) and OCR recognition engine (OcrDetectEngine). VisionService is used to recognize views in the interface, TextConfiguration is used to recognize text objects, and OcrDetectEngine is used to recognize text in the image, that is, to recognize the text in the image based on the font shape. Since the global collection center receives the interface screenshot sent by AMS, the text in it, including the text in the image in the screenshot, can be recognized by OCR.

[0095] The global collection platform reads the whitelist configuration from cloud storage or other storage modules or memories to guide the collection process.

[0096] In conjunction with Figures 2 and 3, when a user performs a three-finger press operation in interface 104, this press operation will be obtained by the input processor and transmitted to the global collection service center, and then the global collection service center will pull up the global collection service. When the user performs a three-finger swipe operation based on the three-finger press operation, this three-finger swipe operation will be obtained by the input processor and transmitted to the global collection service center, and then the collection service module of the global collection service center collects the content in the interface, requests the node from the AMS, and displays the interface 105 through the activity instance. When the user clicks on content extraction in interface 105, the input processor obtains this click operation and transmits it to the global collection service center. The various extraction engines / extraction modules and the intelligent recognition platform in the global service center are linked, and combined with the received whitelist configuration, the corresponding engine / module is called according to the type of each view in interface 105 to perform content extraction, thereby determining the display result shown in interface 106. When the user performs the operation of pressing and sliding to select content in interface 106, the input processor obtains this operation and transmits it to the global collection service middle platform. The global collection service middle platform marks the selected content. When the user clicks the collection control in interface 107, the input processor obtains the click operation and sends it to the global collection service middle platform. The global service middle platform stores the element (material) in the corresponding location and adds it to the collection space according to the view type corresponding to each element in the selected content. Taking the text view as an example, the global collection service middle platform sends the text material directly to the collection space, and displays and stores it in the collection space. When taking the picture view as an example, the global collection service platform stores the picture in the gallery, and sends the thumbnail of the picture to the collection space for storage, and the two are associated. Other situations are not listed one by one. For example, offline web pages can be stored in notes, and only thumbnails are displayed in the collection space; files can be stored in the file manager, and only file information is displayed in the collection space, etc.

[0097] Figure 7 is a schematic diagram of another cross-application collection material processing framework according to an embodiment of the present application. Figures 6 and 7 can be understood as describing the execution process of the solution according to an embodiment of the present application from the perspective of the internal execution process of an electronic device. Figure 6 is an example of the first processing framework, and Figure 7 is an example of the second processing framework.

[0098] The application process includes a view tree and a view recognizer. The view recognizer includes local strategies, recognizers, and stubs. The view tree includes the tree relationship between all subclass views and parent class views. Root refers to the root view. Under the root view are text views, image views, and other subviews, such as file views, web page views, etc., which are not listed one by one. Under the subviews, there can also be custom views, which are not expanded here. These subviews can be identified by the recognizers module in the view recognizer. The identification can include identifying the type, size, whether it is blocked, etc. of the view. Local strategies, after determining the view type of the subview, send the corresponding processing strategy of the subview to the window management service (WindowManagerService, WMS) in the native system service framework (SystemServer) to guide how to display different views when displayed in the collection space. Stubs are used to store relevant data. The view recognizer can also send information such as the type of the identified sub-view and the sub-view to the content recognition service (content recognizer server) in the self-developed system service framework (HnSystemServer).

[0099] The recognizer management function in HnSystemServer triggers the content recognition service, triggered by the Collection Space app and Any Door app. The strategy management function in HnSystemServer manages the processing strategy in the view recognizer. The dump provider determines the storage location and mapping elements for the extracted content of each subview based on the view type. The configuration management function in HnSystemServer determines the extractable content during content extraction based on the received whitelist and / or blacklist configurations. The whitelist is used to customize custom controls in the application. For example, as can be seen in interface 106, the focus control in application A is not extractable content, but it can be made extractable content through the whitelist. It can be understood that the whitelist configuration is used to determine which interfaces can be extracted and / or which views within an interface can be extracted. The blacklist configuration is used to determine which interfaces cannot be extracted and / or which views within an interface cannot be extracted. The basis for non-extraction can be set as needed, meaning that the blacklist configuration can be determined according to actual needs.

[0100] The file drag and drop module in WMS is used to transfer files. For example, text views are stored in the collection space, image views are stored in the gallery, and files are stored in the file manager. Other cases are not listed here.

[0101] Figure 7 illustrates the execution process of a drag operation, illustrating the internal execution of the collection method shown in Figures 1 and 5. During a drag operation, the press position (coordinates) can be retrieved during a long press, and the subviews within the selected content can be traversed and found using the view tree.

[0102] FIG8 is a schematic flow chart of a method for collecting materials across applications according to an embodiment of the present application.

[0103] S801: Run a first application on an electronic device and display a first interface.

[0104] The electronic device includes a first processing framework and a second processing framework. Figures 6 and 7 respectively show examples of the first processing framework and the second processing framework.

[0105] In the embodiment of the present application, the main purpose is to realize the collection of content in the first application into the second application. The first application can be a different application, and the second application is an application that can be globally collected. The so-called global collection means that the content of many applications in the entire electronic device can be collected together, that is, collected in the second application. Application A above is an example of the first application, but it should be understood that the first application can also be other applications such as chat applications, email applications, shopping applications, browser applications, etc., which are no longer listed one by one. Interface 101 and interface 104 can both be regarded as an example of the first interface.

[0106] The collection space mentioned above can be regarded as an example of the second application.

[0107] S802. In response to a first operation of the user on the first interface, the first material in the first interface is added to the second application using the first processing framework, where the first operation is a three-finger swipe down operation in the first interface; or, in response to a second operation of the user on the first interface, the second material in the first interface is added to the second application using the second processing framework, where the second operation is a long press and drag operation on the second material selected in the first interface.

[0108] The first material may correspond to part or all of the content in the first interface.

[0109] In the embodiments of the present application, the subsequent processing procedures for the long-press drag operation and the three-finger swipe operation are different, and different processing frameworks are called for different processing. The drag operation in interface 103 in Figure 1 can be regarded as an example of the second operation. The three-finger swipe operation in interface 104 in Figure 2 can be regarded as an example of the first operation.

[0110] In one implementation, step S802 may include: calling the interface of the AMS in the first processing framework to obtain a screenshot of the first interface; calling the global collection middle platform of the first processing framework to determine the view type contained in the screenshot of the first interface; displaying a second interface on the electronic device, the second interface including a screenshot of the first interface and at least one control, the at least one control being determined based on the view type contained in the screenshot of the first interface; in response to a user clicking on a first control in the at least one control, using the first processing framework to add the first material to the second application in the form of a collection corresponding to the first control. In this implementation, cross-application collection is triggered by a three-finger swipe down operation, and it is necessary to first obtain a screenshot of the interface, determine the view type therein, and then provide the user with corresponding controls to instruct the user in what form to collect the first material, and then collect it in accordance with the collection method indicated by the clicked control according to the way the user clicks the control.

[0111] Interface 105 can be considered as an example of the second interface. The "content extraction", "webpage collection", and "screenshot collection" controls in interface 105 can be considered as an example of at least one control.

[0112] In one implementation, when the first control is a content extraction control, in response to a user clicking operation on a first control in at least one control, using a first processing framework to add the first material to the second application in the form of a collection corresponding to the first control, it can include: using a global collection middleware to traverse and filter all views in the screenshot of the first interface to obtain a first subview set, the first subview set including at least one subview from which content can be extracted; using the global collection middleware, for each subview in the first subview set, performing content extraction according to the view type of the subview to obtain the extracted content corresponding to the subview; when all subviews in the first subview set are extracted, displaying a third interface on the electronic device, the third interface including the extracted content of the first subview set; in response to a user selecting operation on a target content in the extracted content of the first subview set in the third interface, determining the selected target content as the first material; in response to a user clicking operation on a confirmation collection control in the third interface, using the global collection middleware to add the first material to the second application. In this implementation, the user clicks the content extraction control, triggering the extraction of the entire screenshot's content. The process first traverses and filters all subviews in the screenshot with extractable content, then extracts the content of each subview based on its type. All extracted content is then displayed on a third interface for the user to select and add to favorites based on their selections. It should be understood that in this implementation, since it's unknown whether the user wishes to save the entire page or just a portion, or which portion they want to save, the content of all subviews in the entire screenshot is first extracted and made selectable, allowing the user to select which portion to save.

[0113] In one example, in the above implementation, during traversal, for each view in the screenshot of the first interface, at least one of the following operations is performed: if the view is a standard view, the view type is obtained using a standard view acquisition method; if the view is a custom view, the view type is determined using the view suffix; and the size and / or position of the view is read. In this example, some or all of the view type, size, and position information is obtained during traversal, and different acquisition methods are used depending on whether the view is a standard view or a custom view.

[0114] In another example, in the above implementation, during filtering, at least one of the following operations may be performed: retaining only views visible to the user in the screenshot of the first interface, removing hidden views in the screenshot of the first interface, and removing views that are partially or fully obscured in the screenshot of the first interface. In this example, views are filtered to a certain extent. In addition to views visible to the user, an interface may also contain hidden views or other views that are not displayed in the interface but belong to the interface. Therefore, it is possible to retain only views visible to the user or remove hidden views, both of which have equivalent effects. In addition, partially or fully obscured views may also be removed.

[0115] In this example, when removing partially or fully obscured views, the decision to remove the view can be further made based on the percentage of the obscured portion. Specifically, if the percentage of the obscured portion of the view within the entire view is greater than or equal to a preset percentage threshold, the view is removed. This approach facilitates content extraction for views that are obscured but not severely obscured, resulting in a high concentration of useful content.

[0116] In the above example, different views will be identified using different methods. For standard views, it is the view definition method that is standardized in the field, so the types of these views can be obtained according to the standardized acquisition method in the field. For views customized by individual manufacturers, the view type can be determined according to the suffix set during customization. For example, the suffix of textview can be used to determine that it is a text type view, and the suffix of imageview can be used to determine that it is an image type view, etc., which will not be listed one by one. In addition, due to the limitation of screen size, some views cannot be presented in the interface. For example, in interface 104, the news content may be very long and not fully displayed on this page. In this case, when identifying the view, only the view visible to the user, that is, the view within the interface, needs to be identified. For example, at the boundary of the screen, some views may be partially blocked. In this case, these views also need to be screened out. For example, suppose there is a picture at the bottom of interface 104 that is only half displayed. The view corresponding to this picture is the partially blocked view.

[0117] Interfaces 106 and 107 can both be considered examples of the third interface. The white-background content in interface 106 can be considered an example of the extracted content (selectable content) of the first subview set. The press-and-slide selection operation in interface 106 can be considered an example of selecting target content. The dark-gray-background content in interface 107 can be considered an example of target content.

[0118] As can be seen from the above, in the collection process triggered by the three-finger swipe operation, after determining the content to be collected, the user only needs to click the confirm collection control in the third interface to add the content to be collected to the second application.

[0119] The "Collect" controls in interfaces 106 and 107 can be considered as examples of the aforementioned confirmation collection controls. The click operation on the "Collect" control in interface 107 in FIG3 can be considered as an example of the click operation on the confirmation collection control in the third interface.

[0120] In another example, during traversal, for each view, first determine whether it is a standard view; if it is a standard view, obtain its view type using the standard view acquisition method; if it is not a standard view, determine whether the webview is a custom view; if it is a custom view, use the view suffix to determine its view type; if it is neither a standard view nor a custom view, do not add the view to the first subview set. In this example, the priority is set, first checking whether it is a standard view, then checking whether it is a custom view if it is not, and then discarding the view if it is neither a standard view nor a custom view. This allows for more precise view identification.

[0121] In one implementation, the first subview set includes subviews of at least one view type selected from text, image, offline web page, file, video, and link; when using the global collection center, for each subview in the first subview set, content is extracted according to the view type of the subview to obtain the extracted content corresponding to the subview, which may include: for each subview, when the view type of the subview is text, calling the text extraction module in the global collection center to obtain the text data of the subview, and determining the obtained text data as the extracted content of the subview; or, for each subview, when the view type of the subview is image, calling the image extraction engine in the global collection center to obtain a bitmap of the image from the subview, and determining the bitmap as the extracted content of the subview, and / or calling the text processing module in the global collection center to extract the extracted content of the subview through optical character recognition. Recognition, OCR) to obtain the text data contained in the image (the execution area of ​​​​OCR corresponds to the image); or, for each sub-view, when the view type of the sub-view is an offline web page, call the web page extraction engine in the global collection center to extract the title and parse the content of the web page to obtain the extracted content of the sub-view, and the extracted content of the sub-view includes the extracted title and the parsed web page content; or, for each sub-view, when the view type of the sub-view is a file, call the file extraction engine in the global collection center to obtain the file path of the file, and determine the copy of the file or the file path of the file as the The extracted content of the subview, and the file information of the file is determined as the extracted content of the subview; or, for each subview, when the view type of the subview is video, the video extraction engine in the global collection center is called, and the video identification (ID) written to the accessibility node by the first application is read to identify the playback area of ​​the video, and the screen is recorded, screenshots are taken and / or tags are added to obtain the extracted content of the subview; or, for each subview, when the view type of the subview is link, the online link extraction engine in the global collection center is called, and the link information is obtained according to the uniform resource locator (URL) field carried in the activity or the uniform resource locator (getURL) reading method of the web view webview, and the obtained link information is the extracted content of the subview.In this implementation, different content extraction strategies are used for different types of views, so that the present application solution can simultaneously support content extraction of multiple views, rather than just content extraction of a single view.

[0122] In one example, for each subview, when the view type of the subview is text, calling the text extraction module in the global collection center to obtain the text data of the subview, and determining the obtained text data as the extracted content of the subview, it can include: using the text extraction module to call the intelligent recognition platform, so that the intelligent recognition platform obtains the text data of the subview through OCR (the execution area of ​​​​OCR corresponds to the screenshot of the entire first interface), or, using the text extraction module, obtaining the text data of the subview by reading the information written to the accessibility node by the first application. In other words, for text-type subviews, there are two ways to extract content: OCR recognition or reading from the accessibility node.

[0123] In one implementation, in response to a user clicking on the confirmation collection control in the third interface, using the global collection middle platform to add the first material to the second application, at least one of the following operations may be included: directly adding the text material in the first material to the second application; storing the image material in the first material to the gallery application, and adding the thumbnail of the image to the second application, the image stored in the gallery application is associated with the thumbnail of the image in the second application, and the thumbnail of the image is the mapping element of the image in the second application; storing the offline web page material in the first material to the note application, and adding the brief introduction content of the offline web page to the second application, the offline web page stored in the note application is associated with the brief introduction content of the offline web page in the second application, and the offline web page The introduction content is the mapping element of the offline webpage in the second application; the video material in the first material is stored in the gallery application, and the thumbnail of the first frame image of the video is added to the second application. The video stored in the gallery application is associated with the thumbnail of the first frame image of the video in the second application. The thumbnail of the first frame image of the video is the mapping element of the video in the second application; the file material in the first material is stored in the file management application, and the file information of the file is added to the second application. The file stored in the second application is associated with the file information of the file in the second application. The file information of the file is the mapping element of the file in the second application. The file information includes the icon and file name of the file type; and the link material in the first material is directly added to the second application. In this implementation method, it is given that the collection of materials of different view types (different types of materials) in the first material will be completed according to different adding schemes. Figures 6 and 7 also provide an exemplary introduction to this part of the content, which will not be repeated here.

[0124] In another implementation, the first control is a web page collection control, a link collection control, or a file collection control. In response to a user click operation on the first control of at least one control, using the first processing framework to add the first material to the second application in the collection form corresponding to the first control, it can include: when the first control is a web page collection control, adding the first material to the second application in the form of an offline web page, and the first material is the content of the entire web page displayed in the first interface; or, when the first control is a web page collection control, adding the first material to the second application in the form of an offline web page, and the first material is the content displayed on the interface of the web page displayed in the first interface; or, when the first control is a link collection control, adding the first material to the second application in the form of a link, and the first material is an online link of the web page displayed in the first interface; or, when the first control is a file collection control, adding the first material to the second application in the form of a file, and the first material is the file displayed in the first interface, and the file is an open document, a full-screen video, or a full-screen picture. In this implementation, examples of other collection forms are given, which enriches the form of collection content and makes it easier for users to collect according to different needs. In other words, users can collect the content they want to collect in different ways according to different collection needs.

[0125] In another implementation, when calling the global collection middle platform of the first processing framework to determine the view type contained in the screenshot of the first interface, it can include: determining the view type contained in the screenshot of the first interface according to the whitelist configuration of the first application or the whitelist configuration of the first interface. In this implementation, the view type contained in the screenshot is determined according to the whitelist configuration. The introduction of the whitelist configuration can help determine the extractable range in the interface, so that the extracted content adds customized constraints, and different whitelists can be set for different applications or different interfaces as needed.

[0126] In another implementation, when a second interface is displayed on an electronic device and includes a screenshot of the first interface and at least one control, it may include: when the view type included in the screenshot of the first interface includes a web page, at least one control includes at least one of a web page collection control, a link collection control, and a content extraction control; or, when the view type included in the screenshot of the first interface includes a file, at least one control includes at least one of a file collection control and a content extraction control; or, when the view type included in the screenshot of the first interface includes only pictures or videos, at least one control includes at least one of a file collection control and a content extraction control. In this implementation, the type of control displayed on the second interface will be determined based on the different view types included in the screenshot, so that the user can choose as needed. For example, assuming that the first interface does not include a web page view, there is no need to display a web page collection control on the second interface.

[0127] It should be understood that the view types contained in the screenshot of the first interface refer to the view types that are allowed to be extracted in the first interface. For example, for interface 104, what is displayed therein is an online web page, and it does not include videos or files, so the view types that are allowed to be extracted in the first interface include offline web pages, online links, texts and pictures. In other words, the view types contained in interface 104 include offline web pages, online links, texts and pictures. Assuming that a full-screen picture is displayed in the interface, the view types that are allowed to be extracted in the interface include pictures. Assuming that the picture also includes text, the view types that are allowed to be extracted in the interface also include text. Other situations are not listed one by one.

[0128] In another implementation, in response to the user's second operation on the first interface, using the second processing framework to add the second material in the first interface to the second application can include: using the content recognition service in the second processing framework to obtain the second material selected by the user in the first interface; using the content recognition service to determine the view type contained in the second material based on the long press coordinates of the second operation; using the view identifier in the second processing framework to traverse the view tree to obtain the subviews of the extractable content contained in the second material, and obtain a second subview set, which includes at least one subview of the extractable content; using the content recognition service to extract content from each subview in the second subview set according to the view type of the subview, and obtain the extracted content corresponding to the subview; when all subviews in the second subview set are extracted, the extracted content of the second subview set is obtained; using the WMS in the second processing framework to add the extracted content of the second subview set to the second application based on the view type contained in the extracted content of the second subview set. In this implementation, a global collection is triggered by a drag operation. Since dragging requires selecting an object before dragging, and the coordinates of the starting point of the drag (the coordinates when long pressing) can be obtained, the subview of the selected content is determined by traversing the view tree to find all subviews, rather than identifying all subviews in the entire interface. This implementation method first allows the user to select the object (second material) they want to collect, and then performs view identification and content extraction in the view on the second material, and collects the extracted content. The drag operation on interface 103 in Figure 1 can be regarded as an example of a long press and drag operation. Figure 7 gives an example of the internal execution process of this implementation method and an example of the second processing framework.

[0129] In one example, the second subview set includes subviews of at least one view type selected from the group consisting of text, image, offline webpage, file, video, and link, that is, at least one of textview, imageview, webview, fileview, videoview, and onlinelinkview.

[0130] When using the content recognition service to extract content from each subview in the second subview set according to the view type of the subview to obtain the extracted content corresponding to the subview, it can include: for each subview, when the view type of the subview is text, obtaining text data by means of OCR (the execution area of ​​OCR corresponds to the second material rather than the first interface), or, obtaining the text data of the subview by reading the information written to the accessibility node by the first application, and determining the obtained text data as the extracted content of the subview; or, for each subview, when the view type of the subview is an image, obtaining the bitmap of the image from the subview, and determining the bitmap as the extracted content of the subview, and / or, obtaining the extracted content of the subview by means of OCR. The text data contained in the image is obtained in a manner (the execution area of ​​OCR corresponds to the image); or, for each subview, when the view type of the subview is a file, the file path of the file is obtained, and the copy of the file or the file path of the file is determined as the extracted content of the subview; or, for each subview, when the view type of the subview is a video, the video playback area is identified by reading the video ID written to the accessibility node by the first application, and the screen is recorded, screenshots and / or tags are added; or, for each subview, when the view type of the subview is a link, the link information is obtained according to the URL field carried in the activity frame activity or the getURL method of the webview, and the obtained link information is the extracted content of the subview. In this example, under the triggering of the second operation, there will be different content extraction strategies for different types of views, so that the present application solution can simultaneously support content extraction of multiple views, rather than just content extraction of a single view.

[0131] It should be understood that during the drag operation, there is generally no need to collect offline web pages. For example, in the interface 104, the entire web page cannot be selected and then dragged.

[0132] It should be noted that the rules for traversing and filtering subviews and the whitelist configuration mentioned above also apply here and will not be repeated here.

[0133] In one implementation, when using the window management service in the second processing framework, according to the view type contained in the extracted content of the second subview set, the extracted content of the second subview set is added to the second application, which may include at least one of the following operations: directly adding the text content in the extracted content of the second subview set to the second application; storing the image content in the extracted content of the second subview set to the gallery application, and adding the thumbnail of the image to the second application, the image stored in the gallery application is associated with the thumbnail of the image in the second application, and the thumbnail of the image is the mapping element of the image in the second application; storing the offline web page content in the extracted content of the second subview set to the note application, and adding the introduction content of the offline web page to the second application, the offline web page stored in the note application is associated with the thumbnail of the offline web page in the second application. The introduction content is associated, and the introduction content of the offline webpage is the mapping element of the offline webpage in the second application; the video content in the extracted content of the second subview set is stored in the gallery application, and the thumbnail of the first frame image of the video is added to the second application. The video stored in the gallery application is associated with the thumbnail of the first frame image of the video in the second application, and the thumbnail of the first frame image of the video is the mapping element of the video in the second application; the file content in the extracted content of the second subview set is stored in the file management application, and the file information of the file is added to the second application. The file stored in the second application is associated with the file information of the file in the second application. The file information of the file is the mapping element of the file in the second application, and the file information includes the icon and file name of the file type; and the link content in the extracted content of the second subview set is directly added to the second application. In other words, the above method of storing and displaying in the second application according to different material / content types is applicable to this step under both triggering operations.

[0134] In one example, the above process of identifying views and extracting content from sub-views may include: confirming the application page content extraction whitelist through offline testing and investigation (that is, predefining the whitelist configuration according to actual needs), and preliminarily locking the scope of the extractable content of the current page (that is, determining the extractable view in the page); if it is a global collection triggered by a long press and drag operation, traversing the view tree to find all sub-views of the extractable content with the coordinates of the current long press; if it is a three-finger swipe down, determining the scope of the extractable content according to the priority based on the whitelist configuration, and traversing and filtering the views of the extractable content in the current page at the same time, and obtaining the sub-view of the extractable content as needed; based on the sub-View of the extractable content obtained in the previous step, for each type of view, according to multiple sets of different Content extraction is performed according to the rules. If text is involved, OCR capabilities can be used to recognize the text on the current page or read the information written by the application to the accessibility node; if a web page is involved, readabilty is used to extract the title and parse the content; if a video is involved, the video ID written by the application in the accessibility node is read to automatically identify the video playback area, and perform screen recording / screenshots / tags, etc.; if an image is involved, the image bitmap is obtained from the subview to generate the image; if a file is involved, the file path is obtained and copying or path mapping is performed as needed; if an online link is involved, the link is obtained according to the URL field carried in the activity or the web view (webview) read uniform resource locator (getURL) method.

[0135] In one implementation, the method shown in FIG8 further includes steps S803 and S804.

[0136] S803, displaying a snackbar on the electronic device, wherein the snackbar includes a prompt indicating that the collection is successful and a viewing control;

[0137] S804: In response to the user clicking the view control, start the second application, and display the result of adding the content to be collected in the interface of the second application.

[0138] The solution shown in Figure 8 employs two processing frameworks, enabling cross-app collections to be invoked based on different user actions. This makes the collection process more flexible and diverse, meeting diverse user collection needs and facilitating the collection of different types of content on demand. Designing two separate processing frameworks for each of the two triggering actions significantly improves processing efficiency.

[0139] The following is the content of another case.

[0140] As mentioned above, when extracting interface materials, you can first determine the type of view for each view and then extract based on the type of view, but the material types are complex and diverse, and different materials will also have different characteristics. The materials that need to be extracted in the examples of Figures 1-3 include views in various forms such as web pages, online links, texts, and pictures, and instructions are given for the collection of materials such as files and videos in other collection scenarios. How to extract various types of materials is particularly important. Therefore, in order to improve the extraction efficiency, this application also provides a method for extracting page materials, which first aggregates all views of different material types and then extracts each type of material separately.

[0141] Figure 9 is a schematic flow chart of a method for extracting page material according to an embodiment of the present application. It can be seen that the method shown in Figure 9 is an important part of the global collection in Figures 1-3, and helps implement the functions shown in Figures 1-3. The following describes the various steps shown in Figure 9.

[0142] S401: Acquire a first object from a first page.

[0143] The first page is the display interface of the first application running on the electronic device, and the first object contains a view of the material to be extracted. The first application can be, for example, application A shown in Figures 1 to 3. The first page can be, for example, interface 101 and interface 104. The first object can be, for example, the selected floating text + picture shown in interface 103. The first object can also be, for example, a screenshot of the interface shown in interface 105. The first object can also be the entire web page corresponding to interface 104 and / or its online link. Other situations are not listed one by one.

[0144] S402: Determine a material type set of the first object.

[0145] The material type set includes at least one material type of text, picture, file, web page, video and online link.

[0146] The material type refers to the form in which you want to extract the content. It is different from the view type. The former focuses on the content, while the latter focuses on the carrier form of the content, or the form in which the content is presented.

[0147] Taking Figure 1 as an example, the first object is a selected text paragraph and an image. The material type corresponding to the text paragraph is text, and the material type corresponding to the image is image. Taking Figure 2 as an example, the first object is the entire interface 104. Material types include text, images, web pages, and online links. Text materials include the text content in interface 104, images include images in interface 104, web page materials include news web pages, and online link materials include online links within this web page. Other cases are not listed one by one.

[0148] S403: Determine a view set corresponding to each material in the material type set in the first object.

[0149] Each material corresponds to a view set, and each view set includes all views that can extract materials of the material type corresponding to the view set.

[0150] It should be understood that material type and view type are two different concepts. Material type refers to the form of the content you want to save, such as saving it as text, pictures, videos or files, etc., but view type refers to the carrier form of different content when it is displayed on an electronic device. For example, an interface may include text, pictures and videos. The carrier that carries these texts, pictures and videos in the electronic device is called view.

[0151] For example, for webview, it may include text, pictures, videos, online links and other materials. For picture materials, picture view will contain picture materials, file view can also contain materials, and webview can also contain picture materials. Therefore, in the embodiment of the present application, based on the purpose of extracting page materials, considering that the types of carriers (that is, views) of different materials are also different, the material type contained in the selected object (first object) is first determined, and then all views that contain this material are found from the first object for each material type.

[0152] S404: For each material type in the material type set of the first object, extract the material of the material type from the view set corresponding to the material type.

[0153] In one implementation, when the material type set of the first object includes text material, the view set corresponding to the text material includes at least one of text view, image view, file view and webview; when extracting the material of each material type in the material type set of the first object from the view set corresponding to the material type, it may include: when extracting the text material from the view set corresponding to the text material, based on the acquisition method of the first object, extracting the text data in each view in the view set, and determining the extracted text data as the text material extracted from the first object. In this implementation, for the text material, the text view, image view, file view and webview containing text data will be found from the first object and added to its view set. When extracting, the extraction will be based on the acquisition method of the first object, because the different acquisition methods of the first object will play a certain role in determining the content contained in the view set.

[0154] In one implementation, when extracting text material from a view set corresponding to the text material, extracting text data from each view in the view set based on the acquisition method of the first object, and determining the extracted text data as the extracted text material may include:

[0155] When the first object is obtained by a long-press and drag operation on the first object, the view set corresponding to the text material includes a text view and also includes a file view and / or a webview, and text data in the text view is extracted according to a preconfigured extraction rule in a configuration file corresponding to the first object, or, when there is no corresponding configuration file for the first object, text data in the text view is extracted according to a custom extraction rule; or,

[0156] When the first object is obtained by a three-finger swipe down operation in the interface, the view set corresponding to the text material includes an image view of the interface screenshot of the first page, and text is extracted from the image view by optical character recognition (OCR).

[0157] In this implementation, for a long-press drag operation, since the coordinates of the long-press can be determined, and the case where text can be dragged out only includes dragging after selecting a text paragraph in a normal interface, file, or webpage display interface, when a long-press drag operation is performed, the view collection must include the text view and may also include the file view or webview corresponding to the first interface. In this case, the text data in the text view can be extracted according to preconfigured or customized extraction rules. It should be understood that when a text paragraph is selected in a normal interface, an open file, or an open webpage, it is equivalent to the file view and webview containing the text view. Since the first object is dragged after it has been selected, it will only be a portion of the first interface, not the entire interface. Therefore, when extracting text material in a long-press drag operation, it is not necessary to consider the view corresponding to the entire interface, but only the text view within the selected content. For a three-finger swipe operation, since it is an operation on the entire page and no content is pre-selected, the first object may correspond to the entire page, or the entire webpage or file corresponding to the page. Therefore, in a three-finger swipe operation, the text material corresponds to the entire image view of the interface screenshot.

[0158] In another implementation, when the first object does not have a corresponding configuration file, extracting text data from the text view according to a custom extraction rule may include:

[0159] Read the text data of the text view from the accessibility node (AccessibilityNodeInfo) corresponding to the text view; or,

[0160] When the text data of the text view cannot be read from AccessibilityNodeInfo, the text content is read from the text acquisition interface (getText) corresponding to the text view; or,

[0161] When the text data of the text view cannot be read from AccessibilityNodeInfo and getText, the text view identifier (ViewID) is obtained by directly calling view, and then the text data of the text view is obtained according to the text ViewID; or

[0162] When the text data cannot be read from AccessbilityNodeInfo, getText and ViewID, it is obtained from the view character acquisition interface (getViewString) of the text view by directly calling the view.

[0163] In this implementation, custom extraction rules are some execution strategies sorted by execution priority. First, read from the barrier-free node. This requires relying on the first application to write data to the barrier-free node. Most current applications meet this condition. When encountering a few applications that do not meet this condition or fail to read for other reasons, other ways of reading data are tried in turn. The above strategies are from simple to complex from front to back, and the simplest reading method is prioritized.

[0164] Figure 10 below shows an example of how to obtain text data in a dragging scenario (long press and dragging operation scenario). Figure 11 shows an example of extracting text data through OCR recognition in a three-finger swipe down scenario (three-finger swipe down operation scenario).

[0165] In another implementation, when the material type set of the first object includes a picture material, the view set corresponding to the picture material includes at least one of a picture view, a webview, and a custom view;

[0166] For each material type in the material type set of the first object, extracting the material of the material type from the view set corresponding to the material type may include:

[0167] According to the configuration corresponding to the view name or application name, determine at least one view containing the image from the view set corresponding to the image material, including a standard image view, a webview, and a custom view;

[0168] For a standard image view, obtain a bitmap of the standard image view by reading the image storage interface, and determine the bitmap of the standard image view and the uniform resource identifier (URI) of the bitmap of the standard image view as the image material extracted from the first object;

[0169] For the webview, obtain the bitmap and URI of the image in the webview by injecting a JS script, and determine the bitmap of the webview and the URI of the bitmap of the webview as the image material extracted from the first object;

[0170] For a custom view, copy the bitmap by traversing and searching for key attributes in the custom view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the image material extracted from the first object;

[0171] For a custom image view, when a bitmap cannot be found from the key attributes, the bitmap of the custom image view is drawn, and the bitmap of the custom view and the URI of the bitmap of the custom view are determined as the image material extracted from the first object.

[0172] In this implementation, different extraction strategies are determined and executed based on whether the view is an image view or a standard image view. A standard image view is an image view, and images may also exist in a web view, so they are explained separately. For non-standard image views, that is, custom views, if the bitmap can be found, its bitmap is read; if not, the bitmap of the custom view is directly drawn. Figure 12 shows an example of the image extraction process. The above extraction strategy can extract image materials regardless of whether it is a standard image view, and by using the strategy of directly drawing the bitmap as a backup, it can ensure that images in all views containing images can be extracted.

[0173] In an example, for a standard image view, obtaining the bitmap of the standard image view by reading the storage interface of the image may include: reading the bitmap of the standard image view from the storage node of the standard image view; or, when the bitmap of the standard image view cannot be read, drawing the bitmap of the standard image view.

[0174] In this example, a strategy is presented for searching for a standard image's bitmap, prioritizing reading from the storage node. If the bitmap is not found, the bitmap is directly drawn. This strategy provides a fallback for directly drawing the bitmap even if the bitmap cannot be read, ensuring that all views containing images can be retrieved.

[0175] In one implementation, when the material type set of the first object includes a file material, the view set corresponding to the file material includes a file view; for each material type in the material type set of the first object, extracting the material of the material type from the view set corresponding to the material type may include:

[0176] When the first object is a file selected on the first page, and the first object is obtained by a long-press and drag operation on the first object, and the view set corresponding to the file material only includes the file view, directly extract the file information of the file view in the first object, and search whether the file exists under the storage path of the file view. If the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object; or

[0177] When the first page is a display page for an open file, the first object is the open file, and the first object is obtained by a three-finger swipe operation in the first page, and the view set corresponding to the file material includes a file view corresponding to the open file, and also includes a text view and / or a picture view, the file information of the open file is searched from the pre-storage space, and the file is copied according to the file path indicated by the read file information, and the URI of the copied file and the read file information are determined as the file material extracted from the first object, or, when the file information of the open file cannot be found in the pre-storage space within a preset time period after the three-finger swipe operation occurs, the file information of the open file is obtained through the field of the activity, and the file is copied according to the file path indicated by the obtained file information, and the URI of the copied file and the obtained file information are determined as the file material extracted from the first object; the pre-storage space includes file information of all detected open files.

[0178] In this implementation, different file view extraction strategies are provided for the dragging scenario and the three-finger swipe scenario. For file collection, in the dragging scenario, one or more files contained in the interface must be selected and dragged. Whether these files are open or not does not matter, so only file information extraction and other operations are required. For the three-finger swipe scenario, an interface (the first page) must be currently displayed, in which files are being opened and displayed. The three-finger swipe is a three-finger swipe on the opened file. At this time, the file view in the first selected object must be the file view corresponding to the opened and displayed file. At this time, it is necessary to find the file information of this file from the pre-stored file information of all open files, and then trace back to this file to copy and extract the URI.

[0179] In one example, when the first object is a file selected in the first page, the first object is obtained by a long press and drag operation on the first object, and the view set corresponding to the file material only includes the file view, the file information of the file view in the first object is directly extracted, and whether the file exists under the storage path of the file view is searched. When the file exists under the storage path of the file view, the file is copied, and the URI of the copied file and the extracted file information are determined as the file material extracted from the first object. It can include: determining whether the first application is connected to the file drag framework; when the first application is connected to the file drag framework, directly obtaining the URI of the open file; or, when the first application is not connected to the file drag framework, scanning the electronic device through the file name of the open file to find the file path of the open file, thereby obtaining the URI of the open file.

[0180] In this example, if the application (the first application) itself supports dragging and dropping files, the URI of the file can be found directly, but if the application does not support dragging and dropping files, it is necessary to traverse the file system to find the file.

[0181] In another implementation, when the material type set of the first object includes web page material, the view set corresponding to the web page material includes webview; for each material type in the material type set of the first object, when extracting the material of the material type from the view set corresponding to the material type, it can include: extracting the title and text content of the webview in the first object by reading and parsing a readable readability file, and determining the extracted title and text content as the web page material extracted from the first object.

[0182] In this implementation, for webview, web page materials in the form of offline web pages are extracted by parsing the content therein.

[0183] In another implementation, when the material type set of the first object includes video material, the view set corresponding to the video material includes webview; for each material type in the material type set of the first object, when extracting the material of the material type from the view set corresponding to the material type, it can include: determining the video playback area in the first page by reading the video configuration information written by the first application in the accessibility node, and performing at least one of recording, screenshoting and adding tags on the playing video to obtain video excerpt content, and determining the obtained video excerpt content and the URI of the video excerpt content as the video material extracted from the first object.

[0184] In this implementation, the video material must be a partial or complete excerpt of the video played in weview or something similar to webview. The excerpt is a process, so it is equivalent to recording and extracting for a period of time. First, lock the video playback area (for example, when the video is not played in full screen on the web page), and then start recording the screen, taking screenshots, etc. to extract a section or all of the video, and you will get the video excerpt material, which can also be called video material.

[0185] It should be noted that in addition to storing video excerpts, you can also store online links to the video. In this case, the online link material is extracted according to the online link extraction method, which does not belong to the extraction of video material. In addition, when transferring video files between chat applications, the video exists in file form. You need to select the video file and then drag it according to the drag and drop scenario. Therefore, the same method can be used to extract file materials, except that the extracted file material is a video.

[0186] You can understand that videos in the collection space are primarily collected by dragging and dropping video files, while video excerpts can be extracted from the video playback page. If a video link is stored in the collection space, the essence of this material is an online link, which is just an online link associated with the video.

[0187] In another implementation, when the material type set of the first object includes online link material, the view set corresponding to the online link material includes at least one of an online link view and a webview; for each material type in the material type set of the first object, the material of the material type is extracted from the view set corresponding to the material type, including: obtaining the online link material according to the uniform resource locator (URL) field carried in the activity module or reading the uniform resource locator (getURL) from the webview.

[0188] In this implementation, online links can be found in the activity's current field or read from the webview. Some views are not webviews but have similar properties and corresponding links, such as the online editing of shared documents that often occurs in work applications. Other scenarios are not listed here.

[0189] In one example, when obtaining online link materials according to the URL field carried in the activity or the getURL method from the webview, it can include: when the webview corresponding to the first page is a standard webview, directly using the getURL method to obtain the online link of the standard webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is a custom webview, matching the name of the webview from the configuration information of the first page, finding the custom webview corresponding to the first page, and then using the getURL method to obtain the online link of the custom webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is neither a standard webview nor a custom webview, obtaining the online link of the webview from the URL field carried in the activity.

[0190] In this example, different methods of obtaining online links are adopted depending on whether the text view is a standard webview, a custom webview, or neither.

[0191] S405: After all view sets corresponding to the material type set of the first object are extracted, the first material corresponding to the first object is obtained.

[0192] That is to say, the first material is the material extracted from the first object.

[0193] It should be understood that although the first pair of images may contain a variety of materials, not all of them will necessarily be extracted. A portion of the materials may be selected for extraction based on the user's selection. In particular, in the three-finger swipe down scenario, where no object is explicitly selected before extracting the materials, the first object may correspond to part of the content in the first interface, the entire content, or even the webpage or file corresponding to the first interface.

[0194] In one implementation, the method further includes: in response to a user clicking operation on an extraction control, selecting a material type corresponding to the extraction control in the first object, and adding the material type to the material type set.

[0195] In one implementation, the method further includes: sharing the first material to a second application. That is, the extracted material can be shared to an application for management, collection, or use. The second application can be, for example, a collection space.

[0196] The method shown in Figure 9 identifies views within the selected object that contain the type of material they contain, then extracts the material based on the type, meeting diverse material extraction needs. The extracted page materials can then be used to save, merge documents, or share them with other apps or devices.

[0197] It should also be understood that Figure 9 illustrates a solution that first determines the type of material to be extracted from the selected object and then extracts different materials for different views. Alternatively, one could first determine all view types within the selected object and then extract materials from each view within each view type. However, this approach also requires determining which materials can be extracted from each view and which materials need to be extracted. This is relatively straightforward for views like text and image views, which contain only one type of material. However, for file and web views, which are often composite views (e.g., a file view may further include text and image views, or even video views), extracting materials from these views still requires applying different extraction methods for different materials in different views. This approach can result in redundant extractions. Furthermore, because the extraction strategy determines the extraction frameworks to be invoked based on the material type and then applies different extraction strategies based on the views within each extraction framework, this approach requires constantly switching between different extraction frameworks to extract different materials from the same view until extraction is complete. However, when extracting a material from all views where the material exists, the same extraction framework will be called, but different extraction strategies may be used for different views after calling the same framework.

[0198] Figure 10 is a schematic diagram of a page material extraction framework in a long-press drag operation scenario according to an embodiment of the present application. Figure 10 can be seen as an example of an execution framework for extracting page materials under the triggering of a long-press drag operation, or in a long-press drag scenario (or drag scenario).

[0199] As shown in Figure 10, the execution framework includes an application process and a system server process. The system server process primarily provides framework services such as background resources for the execution process within the application process. The application process includes an event interception process, a control classification process, an information extraction process, an information transmission process, and an information storage process.

[0200] The time interception process mainly uses the activity window (activity-window) to intercept drag events. In practice, after the event sensing module (eventHub) in the system service process receives the input event reported by the TP, it reports it to the input dispatch module (InputDispatcher) through the input reading module (InputReader). The window management service (WindowManagerService, WMS) manages the input dispatch module, and the InputDispatcher reports the input event to the application process. The activity-window in the application process intercepts the input event, where the input event is a long-press drag event. In the case of a long press, the coordinates of the long-press position (down(X, Y)) can be detected. The activity-window also records the coordinates of the long press through the long-press coordinate recording module.

[0201] The control classification process primarily determines the view classification and the types of assets contained within each view. In the Android system, views include views corresponding to assets such as text, images, files, and web pages (WebView). Text assets primarily exist in editable text (EditText), text views (TextView), and custom views. Image assets primarily exist in standard image views (ImageView) and non-standard image views, such as custom views and WebViews. Files may contain combined views, such as a text view (TextView) and an image view (ImageView). View type identification can be performed based on the file suffix and control type. WebViews may also contain text and / or image assets. WebViews can obtain information from these assets through JavaScript injection. Other assets primarily include video assets and online link assets. Video assets primarily exist in WebViews, file views, and custom views, while online link assets primarily exist in WebViews and custom views.

[0202] The information extraction process includes text extraction, image extraction, and file extraction. It should be understood that in a drag-and-drop operation, when dragging one or more images to a collection, these images exist as files, so it is equivalent to dragging and dropping files. Dragging and dropping an entire video file is also equivalent to dragging and dropping a file. For example, in a chat app, if a video file is shared with each other after downloading, long-pressing the video thumbnail and dragging it will drag and drop the entire video file as a file.

[0203] When extracting text, you can use AccessbilityNodeInfo, getText, getViewID, or the view character storage interface (getViewString). When extracting images, you can get the image's bitmap. For file extraction, if there is a file drag framework, you can directly get the file path. If there is no file drag framework, you can get the file's uniform resource identifier (URI) by scanning the file name to find the file path. When extracting other materials, you can lock the view collection of the material according to a custom extraction strategy, and then extract the material from it.

[0204] When extracting text, if a third-party application writes text data to AccessbilityNodeInfo, you can retrieve the data directly from it. If not, you can retrieve it from other APIs that store editable text data, such as getText. If only viewable text exists, you can retrieve the text data by finding the view identifier of the text view and using that viewID. For viewable text, if the data is still not found even with the viewID, you can retrieve the text data from getViewString. In practice, different applications have their own characteristics and configuration parameters, typically configuration files. Therefore, you can also use these configuration files to determine how to retrieve text data. For example, if the configuration file contains text data written to AccessbilityNodeInfo, you can retrieve the text data directly from AccessbilityNodeInfo without trying other methods. Alternatively, if the configuration file contains text data written to getText but not to AccessbilityNodeInfo, you can retrieve the text data directly from getText without trying other methods. Other scenarios are not listed here.

[0205] When transmitting information, text information is directly written into the clipboard base class (ClipData), and other materials such as pictures and files are written into ClipData with their URIs.

[0206] The extracted materials can be stored separately through the binder communication channel according to different storage strategies corresponding to different materials. For example, text materials can be directly stored in the storage space of the collection space, and image materials can be stored in the storage space of the gallery.

[0207] Private storage and public storage refer to the storage restrictions set for some applications. The storage content is not shared with other applications and belongs to the private storage of the application. Public storage refers to the storage space that can be used by all applications.

[0208] Regarding file extraction, it should be understood that during the drag-and-drop operation, it is already known that the file being dragged is the file's mapping link, rather than dragging the file on the file display page after the file is opened. Therefore, the file can be directly extracted based on the file path of the file without having to monitor whether the file has been opened.

[0209] FIG11 is a schematic diagram of an execution framework for extracting text materials based on OCR in an embodiment of the present application. It should be noted that FIG11 to FIG16 involve extraction frameworks for different materials. For the same modules, the functional introductions of the same modules can refer to each other. For the sake of brevity, the introductions are not repeated in each figure. FIG11 is an example of the execution framework for extracting text materials in a three-finger swipe down scenario.

[0210] As shown in Figure 11, the InputManager is used to detect three-finger presses (used to launch the global Favorites service) and three-finger swipes (used to trigger the global Favorites service). The HwFingersSnapshooter module in the InputProcessor is used to identify these gestures.

[0211] The input processor sends the various operations obtained to the global collection service (global collection middle platform), which includes the collection service (CollectionService) module, sensor manager (SensorManager), activity instance extraction (ContentExtractActivity) module and text extraction module (TexExtractClient).

[0212] Each extraction engine / module in the global collection center extracts activity examples under the trigger of the gesture module, starts and schedules the corresponding process through the activity manager service (AMS), and performs subsequent operations.

[0213] The global collection center can request a node (resquestContentNode) from AMS. Based on the request, AMS passes the activity instance through the activity thread (ActivityThread) feedback node, thereby sending a screenshot of the interface during the three-finger swipe operation to the global collection center.

[0214] That is to say, after three fingers slide down, the global collection center obtains the bitmap and activity instance of the current page screenshot.

[0215] The global collection platform sends initial data, that is, the data to be recognized, to the intelligent recognition platform AIPlugin, which is the bitmap and / or activity instance of the current page screenshot, and receives the recognition results. The intelligent recognition platform includes vision service (VisionService), text extraction (TextConfiguration) and OCR recognition engine (OcrDetectEngine). VisionService is used to recognize views in the interface, TextConfiguration is used to recognize text objects, and OcrDetectEngine is used to recognize text in images, that is, to recognize the text in the image (page screenshot) based on the font shape. Since the global collection center receives the interface screenshot sent by AMS, the text in the interface screenshot and the text in the image in the interface screenshot can be recognized by OCR.

[0216] The global collection center requests OCR recognition from AIPlugin, then sends the interface screenshot to AIPlugin and receives the text recognized by OCR.

[0217] It should also be understood that when extracting the content of an interface screenshot, the main consideration here is the extraction of text materials therein, but in fact the interface can also include pictures. At this time, it is necessary to use the picture extraction engine in the global collection to extract the pictures contained in the interface screenshot, and then combine the extracted pictures and the text data recognized by the above OCR in coordinate order to obtain the content extraction result of the entire interface. The extraction result may include only text or may also include pictures.

[0218] In conjunction with Figures 2 and 3, when a user performs a three-finger press operation in interface 104, this press operation will be obtained by the input processor and transmitted to the global collection service center. The global collection service center will then pull up the global collection service. When the user performs a three-finger swipe operation based on the three-finger press operation, this three-finger swipe operation will be obtained by the input processor and transmitted to the global collection service center. The collection service module of the global collection service center will then collect the content in the interface, request a node from the AMS, and display interface 105 through an activity instance. When the user clicks on content extraction in interface 105, the input processor obtains this click operation and transmits it to the global collection service center. The text extraction module in the global service center will work in conjunction with the intelligent recognition platform to extract content from the interface screenshot, thereby determining the display result of the text shown in interface 106.

[0219] It should also be noted that view recognition + OCR recognition is fundamentally different from the recognition of traditional image edge recognition algorithms. Traditional image edge recognition algorithms (such as ROI detection algorithms) may be able to identify various elements in the image through edges, such as people, objects, and even text, but they cannot recognize the image. In particular, in the embodiment of the present application, after taking a screenshot of the interface, OCR recognition and image view extraction are performed based on the screenshot. However, if an image edge detection algorithm is used, it cannot lock which small images and text are contained in a large image, and can only recognize all possible people, objects, and other targets in the entire large image.

[0220] FIG12 is a schematic flow chart of a method for extracting image materials according to an embodiment of the present application.

[0221] S701: Initiate image content extraction through Activity.

[0222] S702: In the Activity, filter out all views from which images need to be extracted.

[0223] Before extracting image materials, you need to find the view that contains the image, mainly involving standard imageview, webview, and Huawei custom view.

[0224] S703. The Activity sends all views that need to extract images to the image collection module (ImageCollect).

[0225] S704: In ImageCollect, determine the extraction strategy that needs to be adopted when each view extracts images based on the configuration corresponding to the view name or application name.

[0226] That is to say, different types of views can take different approaches when extracting images from them.

[0227] S705 . In ImageCollect, for the standard image view, read the bitmap from its storage node.

[0228] S706 . In ImageCollect, when the bitmap cannot be read, the standard image view is drawn into a bitmap.

[0229] S707. In ImageCollect, when the view is not a standard imageview, draw the view into a bitmap.

[0230] Not a standard ImageView, but a custom ImageView.

[0231] S708. In ImageCollect, when the view is a webview, obtain the bitmap by injecting a JS script.

[0232] The above extraction process first determines whether the view is a standard image view, webview, or custom view, and then extracts it according to the extraction method of different types of image views.

[0233] S709. ImageCollect writes the bitmap to the file system.

[0234] S710: Write a file into the file system and obtain a URI after storage is complete.

[0235] Figure 13 is a schematic diagram of an execution framework for extracting file materials in a three-finger swipe down scenario according to an embodiment of the present application. As shown in Figure 13, after the input processor receives the three-finger swipe down operation, it sends it to the application process according to the whitelist instruction. The Activity module in the application process includes an intent (Intent) module, a data stream (FeedData) module, and a file name (FileName) module to provide file-related data. The path acquisition module in the application process is used to obtain the storage path of the file, and the file transfer module is used to copy the file. When the file is already in the public storage space, it can be omitted. When the file is in the private storage space, in order to prevent the file from being accessed due to subsequent restrictions of the application, the file is copied to the public storage space or the storage space corresponding to the global collection function. The file management service is used to provide services related to file material extraction for the global collection. The file management service includes a file manager (FileManager) for managing files and a file information service (FieldInfoProvider) for extracting file information. Here, the file management service can perform real-time monitoring (detection) of files, that is, detect the opened files, and then send the monitored file information to the global collection center when the file information needs to be viewed in the global collection. The file installation module (Istalled) includes the command line tool Inotify and the file executor (FileExecutor), which are used to provide services for opening, viewing and editing files. It is also used to find hidden public storage paths from the file system and send the hidden public storage paths to the collection module CollectionProvider of the global collection center.

[0236] When triggered by a three-finger swipe operation, the global collection center obtains the file information of the file that is currently open and the file corresponding to the three-finger swipe operation from the file management service. For example, a three-finger swipe operation may be performed on the display interface after file A is opened. The file information is sent to the file system through the collection service CollectionService of the global collection center to find the storage path corresponding to the file system, and the URI of the file is stored in the database (data storage unit) under the CollectionProvider. The global collection center also provides a whitelist for the input processor to guide whether the file material can be extracted.

[0237] In other words, there are three ways to obtain file materials: searching from the monitored file information, directly obtaining through the path acquisition module in the application process, or obtaining after assembly.

[0238] FIG14 is a schematic flowchart of extracting file materials in a three-finger swipe scenario according to an embodiment of the present application.

[0239] S901: The file monitoring module initializes and starts the file monitoring service.

[0240] S902: The file monitoring module starts executing the file monitoring service and monitors in real time whether there are any opened files.

[0241] S903: The user opens any file in the application interface.

[0242] S904: The application interface reports the file opening event to the file monitoring module.

[0243] S905: The file monitoring module obtains file information of the open file.

[0244] S906: The file monitoring module reports the acquired file information to the document extraction framework.

[0245] S907: The document extraction framework saves the file information reported by the file monitoring module.

[0246] In one implementation, the storage here may only store information about files that are currently open and displayed, that is, the files that are currently open and displayed are presented to the user on the application interface. Whenever the user switches application interfaces and the file is no longer displayed on the interface, the file is considered no longer open and displayed and is deleted from the document extraction framework.

[0247] In another implementation, the storage here can also be the file information of all files that are open on the electronic device. In this case, open means being opened on the electronic device, or it can be understood as being in a state that is not closed. For example, two files are opened at the same time, one is displayed in the foreground, and the other is suspended in the background. Both are open files. This method is to prevent users from temporarily viewing other files and then coming back to view the current file. If the previous real-time update is used, the current file will first become unopened and then become opened again, resulting in unnecessary deletion and re-storage of information.

[0248] In another implementation, only file information for files opened within a preset time period from the current moment is stored. For example, only file information for files opened within 10 minutes may be stored. This approach prevents file information for files that the user has not closed from occupying storage space and prevents excessive file information from being stored, resulting in a search requiring multiple files to be searched.

[0249] In another implementation, only file information within a preset threshold value range is stored. For example, only file information of less than or equal to two files may be stored. This implementation restricts the storage of file information based on the storage quantity, so that the stored file information is within a certain range. The preset threshold value range can be set as needed. For example, for scenarios that do not support split-screen operation and cannot display multiple files simultaneously, a relatively small value can be set. For scenarios that support multi-screen display of multiple files, a relatively large value can be set.

[0250] The above-mentioned various implementation methods can be reasonably combined.

[0251] S901-S907 are operations that can be executed all the time, regardless of whether the three-finger swipe down operation is executed. However, in order to prevent file monitoring from being executed in the background all the time, occupying resources and consuming power, a trigger condition can be added before step S901 so that monitoring is only executed under certain conditions.

[0252] In one implementation, the file monitoring is initiated in response to a startup operation of an application having a file viewing and / or editing function. In other words, the file monitoring is performed only when there is a possibility of opening a file.

[0253] S908: The user performs a three-finger swipe down operation on the file A opened in the application interface.

[0254] S909: The application interface reports a file collection task, i.e., a file material extraction task, to the document extraction framework. The document extraction framework retrieves the file information of file A from all stored file information based on the collection task. If the file information of file A cannot be obtained, the framework retrieves the file information of file A from the activity field.

[0255] That is to say, in the three-finger swipe down operation, the operation must be performed in the display interface of the open file, so the stored file information is searched first, and if no file is found, the activity is searched.

[0256] It should also be understood that since the file in step S903 is an arbitrary file, it may not be file A in S908, and when three fingers slide down in S908, it means that file A has executed step S903, but it is unknown whether steps S903-S907 have been completed. Therefore, when step S910 starts to execute, step S907 corresponding to file A has not been completed, that is, the file information of file A has not been stored, and it may be that the file information of file A cannot be queried in step S910.

[0257] Figure 15 is a schematic diagram of an execution framework for extracting web page materials in an embodiment of the present application. As shown in Figure 15, the three-finger swipe operation obtained from the InputManager is sent to the global collection service. The CollectionService in the global collection service distributes the three-finger swipe event and sends a web page collection request to the CollectionService in the note application through binder communication. After receiving the web page collection request, the CollectionService in the note application needs to store the web page content. The sensor manager (SensorManager) obtains the URL of the web page and copies the HTML file (mht file) of the web page. The webview module in the note application loads the web page content, performs parsing of the web page content, calls the title and body extraction algorithm in the extraction module to parse the title and body content of the web page, and then refreshes the page to the webview module. The webview module sends the obtained offline web page content and the unique identifier UUID of the offline web page in the note application to the CollectionProvider in the global collection service. The CollectionProvider receives the web page material (including the extracted title, body content and UUID), and then stores the data to be stored in the data storage unit Collections.db in the global collection service according to the storage mechanism of the web page material.

[0258] Figure 16 is a schematic diagram of an execution framework for extracting online links in an embodiment of the present application. It should be understood that the online links in Figure 16 include web links and video links. The three-finger swipe operation obtained from the InputManager is sent to the video excerpt application or application process. When video excerpting is required, the video excerpting module in the video excerpt application performs video excerpting according to similar steps shown in Figure 16. The video excerpting module includes a floating tool window / bar for obtaining user interaction operations, and also includes scene recognition, which can perform operations such as video excerpting, screenshots, and adding notes. The application process side can directly obtain the web link through the connection acquisition module, or assemble it first and then obtain the web link, and then send the web link to the video link in the video excerpt application. The video link module includes a configuration management and category identification module. The activity in the application process includes Intent, video identification (MediaID) and playback module (playProxy).

[0259] The video excerpt application sends the excerpted video to the global collection center, and the global collection center saves the video link or web page link to the file system and obtains the URI of the video excerpt content.

[0260] The AccessibilityService, MediaCodecServer, and WindowManager modules in the Framework (FWK) provide background support for the video excerpting application. The global collection center sends a whitelist to the InputManager.

[0261] It should be understood that for a webpage, which may include a video, a user may want to save the link of the entire webpage, or may only want to save the specific link of a video, or may only want to extract a portion of a video, which is why there is a situation where both the video link and the webpage link exist as shown in Figure 16. For extraction of online links without any video extraction requirements, the part related to the video excerpt in Figure 16 can be removed.

[0262] In one implementation, when the current page is a standard webview, the online link of the webview is directly obtained according to the GetURL method of the webview; or,

[0263] When the current page is not a standard webview, for example, it may be a custom webview, you can match the name of the webview from the configuration information of the current page, then find the webview, and then get the online link of the webview according to the GetURL method; or,

[0264] When the current page is neither a standard webview nor a custom webview, you can read the online link from the activity field.

[0265] The basis for the above judgment (whether it is a webview or some other view) can be found from the whitelist configuration information.

[0266] The above mainly introduces the method of the embodiment of the present application in conjunction with the accompanying drawings. It should be understood that although the various steps in the flowcharts involved in the various embodiments described above are shown in sequence, these steps are not necessarily performed in sequence in the order shown in the figures. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps. The device of the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0267] Figure 17 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. As shown in Figure 17, the device 2000 includes an acquisition unit 2001 and a processing unit 2002. The device 2000 can be an electronic device such as a mobile phone, a tablet computer, or a touch-screen laptop.

[0268] The device 2000 can be used to perform any of the methods described above. For example, the acquisition unit 2001 can be used to perform step S801, and the processing unit 2002 can be used to perform steps S802-S804. For another example, the acquisition unit 2001 can be used to perform step S401, and the processing unit 2002 can be used to perform steps S402-S405. The device 2000 can also be used to perform the interactive processes or execution processes shown in Figures 1-7 and Figures 9-16.

[0269] In one implementation, the device 2000 may further include a storage unit for storing relevant data. The storage unit may be integrated into any one of the above units or may be a unit independent of all the above units.

[0270] FIG18 is a schematic diagram of the hardware structure of another electronic device according to an embodiment of the present application. As shown in FIG18 , the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone jack 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, and a subscriber identification module (SIM) card interface 295. Among them, the sensor module 280 can include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0271] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0272] For example, the processor 210 shown in FIG18 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0273] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0274] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0275] Electronic device 200 implements display functionality through a GPU, display screen 294, and an application processor. A GPU is a microprocessor for image processing that connects display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0276] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the electronic device 200 may include one or N display screens 294, where N is a positive integer greater than one.

[0277] The electronic device 200 can implement a shooting function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, and an application processor.

[0278] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 200, such as image recognition, face recognition, speech recognition, and text comprehension.

[0279] In the embodiment of the present application, OCR recognition can be performed with the help of NPU or GPU.

[0280] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0281] In an embodiment of the present application, the extracted material can be stored in an internal memory or an external memory, or the URI or URL of the material can be stored in the internal memory, and the source file can be stored in the external memory, etc., which are not listed one by one.

[0282] Pressure sensor 280A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 280A can be located on display screen 294. There are many types of pressure sensors 280A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 280A, the capacitance between the electrodes changes. Electronic device 200 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 294, electronic device 200 detects the touch intensity based on pressure sensor 280A. Electronic device 200 can also calculate the touch location based on the detection signal from pressure sensor 280A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0283] The touch sensor 280K is also called a "touch panel." The touch sensor 280K can be disposed on the display screen 294. The touch sensor 280K and the display screen 294 form a touch screen, also called a "touch screen." The touch sensor 280K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 294. In other embodiments, the touch sensor 280K can also be disposed on the surface of the electronic device 200, at a location different from that of the display screen 294.

[0284] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0285] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0286] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and capable of running on the at least one processor, wherein when the processor executes the computer program, the electronic device can implement the steps in any of the above methods.

[0287] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by an electronic device, the steps in the above-mentioned various method embodiments can be implemented.

[0288] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the steps in the above-mentioned various method embodiments can be implemented.

[0289] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, mobile hard disk, magnetic disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0290] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0291] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0292] In the embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0293] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0294] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0295] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0296] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0297] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0298] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0299] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for collecting materials across applications, characterized in that, Applied to an electronic device, the electronic device includes a first processing framework and a second processing framework, and the method includes: Running a first application on the electronic device and displaying a first interface; In response to a first operation of the user on the first interface, using the first processing framework to add a first material in the first interface to a second application, the first operation being a three-finger swipe-down operation on the first interface; or, In response to a second operation of the user on the first interface, using the second processing framework to add a second material in the first interface to the second application, the second operation being a long-press drag operation on the selected second material in the first interface.

2. The method according to claim 1, wherein The step of, in response to a first operation of the user on the first interface, using the first processing framework to add a first material in the first interface to a second application, includes: Invoking an interface of the Activity Management Service (AMS) in the first processing framework to obtain a screenshot of the first interface; Invoking the global collection middleware platform of the first processing framework to determine the view types included in the screenshot of the first interface; Displaying a second interface on the electronic device, the second interface including the screenshot of the first interface and at least one control, the at least one control being determined according to the view types included in the screenshot of the first interface; In response to a click operation of the user on a first control among the at least one control, using the first processing framework to add the first material to the second application in the collection form corresponding to the first control.

3. The method according to claim 2, wherein The first control is a content extraction control. The step of, in response to a click operation of the user on the first control among the at least one control, using the first processing framework to add the first material to the second application in the collection form corresponding to the first control, includes: Traversing and filtering all views in the screenshot of the first interface by using the global collection middleware platform to obtain a first subset of views, the first subset of views including at least one sub-view capable of extracting content; Using the global collection middleware platform to perform content extraction for each sub-view in the first subset of views according to the view type of the sub-view to obtain the extracted content corresponding to the sub-view; After all sub-views in the first subset of views are extracted, displaying a third interface on the electronic device, the third interface including the extracted content of the first subset of views; In response to a selection operation of the user on the target content in the extracted content of the first subset of views in the third interface, determining the selected target content as the first material; In response to a click operation of the user on the confirmation collection control in the third interface, using the global collection middleware platform to add the first material to the second application.

4. The method according to claim 3, wherein The step of traversing and filtering all views in the screenshot of the first interface by using the global collection middleware platform to obtain a first subset of views, includes: When traversing, for each view in the screenshot of the first interface, perform at least one of the following operations: when the view is a standard view, obtain the view type of the view in the way of obtaining a standard view, or, when the view is a custom view, determine the view type of the view by using the suffix of the view; and, read the size and / or position of the view. When filtering, perform at least one of the following operations: only keep the views visible to the user in the screenshot of the first interface, remove the hidden views from the screenshot of the first interface, and remove the views partially or completely blocked in the screenshot of the first interface.

5. The method according to claim 3 or 4, characterized in that, The first subset of sub-views includes sub-views of at least one view type among text, picture, offline web page, file, video, and link; using the global collection middleware platform, for each sub-view in the first subset of sub-views, perform content extraction according to the view type of the sub-view to obtain the extracted content corresponding to the sub-view, including: For each sub-view, when the view type of the sub-view is text, call the text extraction module in the global collection middleware platform to obtain the text data of the sub-view, and determine the obtained text data as the extracted content of the sub-view; or, For each sub-view, when the view type of the sub-view is a picture, call the picture extraction engine in the global collection middleware platform to obtain the bitmap of the picture from the sub-view, and determine the bitmap as the extracted content of the sub-view, and / or, call the text processing module in the global collection middleware platform to obtain the text data contained in the picture by means of optical character recognition (OCR); or, For each sub-view, when the view type of the sub-view is an offline web page, call the web page extraction engine in the global collection middleware platform to perform title extraction and content parsing on the web page to obtain the extracted content of the sub-view, and the extracted content of the sub-view includes the extracted title and the parsed web page content; or, For each sub-view, when the view type of the sub-view is a file, call the file extraction engine in the global collection middleware platform to obtain the file path of the file, and determine the copied file of the file or the file path of the file as the extracted content of the sub-view, and determine the file information of the file as the extracted content of the sub-view; or, For each sub-view, when the view type of the sub-view is a video, call the video extraction engine in the global collection middleware platform, identify the playing area of the video by reading the video identification ID written by the first application to the accessibility node, and perform screen recording, screenshot, and / or add tags, so as to obtain the extracted content of the sub-view; or, For each sub-view, when the view type of the sub-view is a link, call the online link extraction engine in the global collection middleware platform, and obtain link information according to the uniform resource locator (URL) field carried in the activity or the way of getting the URL of the web view. The obtained link information is the content after extraction of the sub-view.

6. The method according to claim 5, wherein For each sub-view, when the view type of the sub-view is text, call the text extraction module in the global collection middleware platform to obtain the text data of the sub-view, and determine the obtained text data as the content after extraction of the sub-view, including: Use the text extraction module to call the intelligent recognition platform, so that the intelligent recognition platform obtains the text data of the sub-view by means of OCR, or use the text extraction module to obtain the text data of the sub-view by reading the information written into the accessibility node by the first application.

7. The method according to any one of claims 3 to 6, characterized in that In response to the user's click operation on the confirmation collection control in the third interface, use the global collection middleware platform to add the first material to the second application, including at least one of the following operations: Directly add the text material in the first material to the second application; Store the picture material in the first material in the gallery application, and add the thumbnail of the picture to the second application. The picture stored in the gallery application is associated with the thumbnail of the picture in the second application, and the thumbnail of the picture is the mapping element of the picture in the second application; Store the offline web page material in the first material in the note application, and add the brief content of the offline web page to the second application. The offline web page stored in the note application is associated with the brief content of the offline web page in the second application, and the brief content of the offline web page is the mapping element of the offline web page in the second application; Store the video material in the first material in the gallery application, and add the thumbnail of the first frame image of the video to the second application. The video stored in the gallery application is associated with the thumbnail of the first frame image of the video in the second application, and the thumbnail of the first frame image of the video is the mapping element of the video in the second application; Store the file material in the first material in the file management application, and add the file information of the file to the second application. The file stored in the second application is associated with the file information of the file in the second application, and the file information of the file is the mapping element of the file in the second application. The file information includes the icon and file name of the file type; And, Directly add the link material in the first material to the second application.

8. The method according to any one of claims 2 to 7, characterized in that The first control is a web page collection control, a link collection control, or a file collection control. Responding to a user's click operation on the first control among the at least one control, adding the first material to the second application in the collection form corresponding to the first control by using the first processing framework includes: When the first control is a web page collection control, adding the first material to the second application in the form of an offline web page, where the first material is the content of the entire web page displayed in the first interface; or, When the first control is a web page collection control, adding the first material to the second application in the form of an offline web page, where the first material is the content displayed on the interface of the web page displayed in the first interface; or, When the first control is a link collection control, adding the first material to the second application in the form of a link, where the first material is the online link of the web page displayed in the first interface; or, When the first control is a file collection control, adding the first material to the second application in the form of a file, where the first material is the file displayed in the first interface, and the file is an open document, a full-screen video, or a full-screen picture.

9. The method according to any one of claims 2 to 8, characterized in that, Invoking the global collection middle platform of the first processing framework to determine the view types included in the screenshot of the first interface includes: determining the view types included in the screenshot of the first interface according to the whitelist configuration of the first application or the whitelist configuration of the first interface; Displaying a second interface on the electronic device, where the second interface includes a screenshot of the first interface and at least one control, includes: When the view types included in the screenshot of the first interface include web pages, at least one of the at least one control includes at least one of a web page collection control, a link collection control, and a content extraction control; or, When the view types included in the screenshot of the first interface include files, at least one of the at least one control includes at least one of a file collection control and a content extraction control; or, When the view types included in the screenshot of the first interface only include pictures or videos, at least one of the at least one control includes at least one of a file collection control and a content extraction control.

10. The method according to any one of claims 1 to 9, characterized in that, Responding to a second operation of the user on the first interface, adding the second material in the first interface to the second application by using the second processing framework includes: Obtaining the second material selected by the user in the first interface by using the content recognition service in the second processing framework; Using the content recognition service to determine the view types included in the second material according to the long-press coordinates of the second operation; Using the view recognizer in the second processing framework to traverse and search the view tree to obtain the sub-views with extractable content included in the second material, obtaining a second sub-view set, where the second sub-view set includes at least one sub-view with extractable content; Using the content recognition service, for each sub-view in the second sub-view set, content extraction is performed according to the view type of the sub-view to obtain the extracted content corresponding to the sub-view; After all sub-views in the second sub-view set are extracted, the extracted content of the second sub-view set is obtained; Using the window management service WMS in the second processing framework, according to the view types included in the extracted content of the second sub-view set, the extracted content of the second sub-view set is added to the second application.

11. The method according to claim 10, wherein The step of using the window management service WMS in the second processing framework to add the extracted content of the second sub-view set to the second application according to the view types included in the extracted content of the second sub-view set includes at least one of the following operations: Directly add the text content in the extracted content of the second sub-view set to the second application; Store the picture content in the extracted content of the second sub-view set in the gallery application, and add the thumbnail of the picture to the second application. The picture stored in the gallery application is associated with the thumbnail of the picture in the second application, and the thumbnail of the picture is the mapped element of the picture in the second application; Store the offline web page content in the extracted content of the second sub-view set in the note application, and add the brief introduction content of the offline web page to the second application. The offline web page stored in the note application is associated with the brief introduction content of the offline web page in the second application, and the brief introduction content of the offline web page is the mapped element of the offline web page in the second application; Store the video content in the extracted content of the second sub-view set in the gallery application, and add the thumbnail of the first frame image of the video to the second application. The video stored in the gallery application is associated with the thumbnail of the first frame image of the video in the second application, and the thumbnail of the first frame image of the video is the mapped element of the video in the second application; Store the file content in the extracted content of the second sub-view set in the file management application, and add the file information of the file to the second application. The file stored in the second application is associated with the file information of the file in the second application, and the file information of the file is the mapped element of the file in the second application. The file information includes the icon of the file type and the file name; And, Directly add the link content in the extracted content of the second sub-view set to the second application.

12. The method according to any one of claims 1 to 11, characterized in that, After adding the first material or the second material to the second application, the method further includes: Display a floating capsule snackbar on the electronic device. The snackbar includes a prompt indicating successful collection and a view control; In response to the user's click operation on the view control, start the second application and display the addition result on the interface of the second application.

13. The method according to any one of claims 1 to 12, characterized in that The first material is obtained by performing the following steps: Obtain a first object of the first interface, where the first object contains a view for the material to be extracted. Determine the material type set of the first object, where the material type set includes at least one of the material types of text, picture, file, web page, video, and online link. Determine the view set corresponding to each material in the material type set in the first object. Each material corresponds to a view set, and each view set includes all views that can extract the material of the material type corresponding to the view set. For each material type in the material type set of the first object, extract the material of the material type from the view set corresponding to the material type. After all the view sets corresponding to the material type set of the first object are extracted, obtain the first material corresponding to the first object.

14. The method according to claim 13, wherein When the material type set of the first object includes text material, the view set corresponding to the text material includes at least one of text view, picture view, file view, and web view; for each material type in the material type set of the first object, extracting the material of the material type from the view set corresponding to the material type includes: When extracting text material from the view set corresponding to the text material, based on the acquisition method of the first object, extract the text data in each view of the view set, and determine the extracted text data as the text material extracted from the first object.

15. The method according to claim 14, wherein The above-mentioned when extracting text material from the view set corresponding to the text material, based on the acquisition method of the first object, extract the text data in each view of the view set, and determine the extracted text data as the extracted text material, includes: When the acquisition method of the first object is a long-press and drag operation on the first object, the view set corresponding to the text material includes a text view and also includes a file view or a web view. Extract the text data in the text view according to the pre-configured extraction rules in the configuration file corresponding to the first object, or, when the first object does not have a corresponding configuration file, extract the text data in the text view according to the custom extraction rules; or, When the acquisition method of the first object is a three-finger swipe-down operation on the interface, the view set corresponding to the text material includes a picture view of the screenshot of the first interface. Extract text from each view in the view set corresponding to the text material by means of optical character recognition (OCR).

16. The method according to claim 15, characterized in that, The above-mentioned when the first object does not have a corresponding configuration file, extract the text data in the text view according to the custom extraction rules, includes: Read the text data of the text view from the AccessbilityNodeInfo corresponding to the text view; or When the text data of the text view cannot be read from the AccessbilityNodeInfo, read the text content from the getText of the text acquisition interface corresponding to the text view; or, When the text data of the text view cannot be read from both the AccessbilityNodeInfo and the getText, obtain the text view identifier ViewID by directly calling the view, and then obtain the text data of the text view according to the ViewID; or, When the text data cannot be read from the AccessbilityNodeInfo, the getText, and the ViewID, obtain it from the getViewString of the view character acquisition interface of the text view by directly calling the view.

17. The method according to claim 13, characterized in that, When the material type set of the first object includes picture materials, the view set corresponding to the picture materials includes at least one of a picture view, a webview, and a custom view; For each material type in the material type set of the first object, extracting the material of the material type from the view set corresponding to the material type includes: Determining at least one view containing a picture among a standard picture view, a webview, and a custom view from the view set corresponding to the picture material according to the configuration corresponding to the view name or the application name; For the standard picture view, obtain the bitmap of the standard picture view by reading the storage interface of the picture, and determine the bitmap of the standard picture view and the uniform resource identifier URI of the bitmap of the standard picture view as the picture material extracted from the first object; For the webview, obtain the bitmap and URI of the picture in the webview by injecting a JS script, and determine the bitmap of the webview and the URI of the bitmap of the webview as the picture material extracted from the first object; For the custom view, copy the bitmap by traversing and finding the key attributes in the custom view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the picture material extracted from the first object; For the custom picture view, when the bitmap cannot be found from the key attributes, draw the bitmap of the custom picture view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the picture material extracted from the first object.

18. The method according to claim 17, wherein For the standard picture view, obtaining the bitmap of the standard picture view by reading the storage interface of the picture includes: Reading the bitmap of the standard picture view from the storage node of the standard picture view; or, When the bitmap of the standard picture view cannot be read, draw the bitmap of the standard picture view.

19. The method according to claim 13, wherein When the material type set of the first object includes file materials, the view set corresponding to the file materials includes file views; for each material type in the material type set of the first object, extracting the material of this material type from the view set corresponding to this material type includes: When the first object is a file selected in the first interface, the acquisition method of the first object is a long-press drag operation on the first object, and the view set corresponding to the file material only includes file views, directly extract the file information of the file view in the first object, check whether the file exists under the storage path of the file view, and when the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object; or, When the first interface is a display page for opening a file, the first object is the opened file, the acquisition method of the first object is a three-finger swipe-down operation in the first interface, the view set corresponding to the file material includes the file view corresponding to the opened file, and also includes text views and / or picture views, search for the file information of the opened file in the pre-storage space, copy the file according to the file path indicated by the read file information, and determine the URI of the copied file and the read file information as the file material extracted from the first object, or, when the file information of the opened file cannot be found in the pre-storage space within a preset time period after the three-finger swipe-down operation occurs, obtain the file information of the opened file through the fields of the activity, copy the file according to the file path indicated by the obtained file information, and determine the URI of the copied file and the obtained file information as the file material extracted from the first object; the pre-storage space includes the file information of all detected opened files.

20. The method according to claim 19, wherein The case where when the first object is a file selected in the first interface, the acquisition method of the first object is a long-press drag operation on the first object, and the view set corresponding to the file material only includes file views, directly extract the file information of the file view in the first object, check whether the file exists under the storage path of the file view, and when the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object, includes: Judge whether the first application is connected to the file drag-and-drop framework; When the first application is connected to the file drag-and-drop framework, directly obtain the URI of the opened file; or, When the first application is not connected to the file drag-and-drop framework, scan and search for the file path of the opened file in the electronic device based on the file name of the opened file, so as to obtain the URI of the opened file.

21. The method according to claim 13, wherein When the material type set of the first object includes web materials, the view set corresponding to the web materials includes webview; for each material type in the material type set of the first object, extracting the materials of this material type from the view set corresponding to this material type includes: By reading and parsing the readability file, extracting the title and body content of the webview in the first object, and determining the extracted title and body content as the web materials extracted from the first object.

22. The method according to claim 13, characterized in that, When the material type set of the first object includes video materials, the view set corresponding to the video materials includes webview; for each material type in the material type set of the first object, extracting the materials of this material type from the view set corresponding to this material type includes: By reading the video configuration information written by the first application in the accessibility node, determining the video playback area in the first interface, and performing at least one of the operations of screen recording, screenshotting, and adding tags to the currently playing video to obtain video excerpt content, and determining the obtained video excerpt content and the URI of the video excerpt content as the video materials extracted from the first object.

23. The method according to claim 13, characterized in that, When the material type set of the first object includes online link materials, the view set corresponding to the online link materials includes at least one of an online link view and a webview; for each material type in the material type set of the first object, extracting the materials of this material type from the view set corresponding to this material type includes: Obtaining the online link materials according to the uniform resource locator URL field carried in the activity or by reading the getURL of the uniform resource locator from the webview.

24. The method according to claim 23, wherein The obtaining the online link materials according to the uniform resource locator URL field carried in the activity or by reading the getURL of the uniform resource locator from the webview includes: When the webview corresponding to the first interface is a standard webview, directly using the getURL method to obtain the online link of this standard webview, and the obtained online link is the online link material extracted from the first object; or, When the webview corresponding to the first interface is a custom webview, matching the name of the webview from the configuration information of the first interface, finding the custom webview corresponding to the first interface, and then using the getURL method to obtain the online link of this custom webview, and the obtained online link is the online link material extracted from the first object; or, When the webview corresponding to the first interface is neither a standard webview nor a custom webview, obtain the online link of the webview from the URL field carried in the activity.

25. An electronic device, comprising a memory, one or more processors, and a computer program stored in the memory and executable on the processors, wherein, When the one or more processors execute the computer program, the electronic device implements the method according to any one of claims 1 to 24.

26. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by an electronic device, the method according to any one of claims 1 to 24 is implemented.

Images