Method for extracting page materials and electronic equipment

Abstract

The invention provides a method for extracting page materials and electronic equipment. The method comprises the steps that material types contained in a selected object are determined, a view set corresponding to each material type is determined, each material is extracted from the view set corresponding to each material type, and all the materials extracted from the selected object are obtained through summarizing. According to the scheme, the view containing the material in the selected object can be locked according to the type of the material contained in the selected object, and then the material is extracted in a classified manner, so that diversified material extraction requirements are met. The extracted page materials can be subsequently used for collecting and merging documents or sharing to other applications or other devices.

Description

Technical Field

[0001] This application relates to the technical field of electronic devices, and particularly to a method for extracting page materials and an electronic device. Background Art

[0002] Collections in traditional electronic devices can only be scattered in different applications, which is inconvenient to search. Moreover, the collected materials can be diverse, such as various types of materials including text, pictures, videos, audios, web pages, and files. In order to facilitate users to manage and use these materials scattered in various applications in different forms, R & D personnel have designed a new application. In this application, users can put all the content they want to collect across applications into it for centralized management, that is, to achieve global collection.

[0003] However, since global collection involves various types of materials, and different materials also have different characteristics, how to better extract different materials in a page is a technical problem to be solved urgently. Summary of the Invention

[0004] This application provides a method for extracting page materials and an electronic device, and this solution can extract different materials in a page.

[0005] In a first aspect, a method for extracting page materials is provided. The method includes: obtaining a first object from a first page, where the first page is a display interface of a first application running on an electronic device, and the first object contains a view of the material to be extracted; determining a set of material types of the first object, where the set of material types includes at least one of the material types of text, picture, file, web page, video, and online link; determining a set of views corresponding to each material in the set of material types in the first object, each material corresponds to a set of views, and each set of views includes all views that can extract the material of the material type corresponding to the set of views; for each material type in the set of material types of the first object, extracting the material of the material type from the set of views corresponding to the material type; after all the sets of views corresponding to the set of material types of the first object are extracted, obtaining a first material corresponding to the first object.

[0006] In the technical solution of this application, the views that will contain such materials in the selected object will be locked according to the material types included in the selected object, and then the materials will be extracted by classification, meeting the diversified material extraction requirements. The extracted page materials can be used for subsequent collection, merging documents, or sharing to other applications or other devices.

[0007] In combination with the first aspect, in certain implementations of the first aspect, when the material type set of the first object includes text materials, the view set corresponding to the text materials includes at least one of a text view, a picture view, a file view, and a web view; when extracting materials of each material type from the view set corresponding to the material type of the first object, it may include: when extracting text materials from the view set corresponding to the text materials, based on the acquisition method of the first object, extracting the text data in each view in the view set, and determining the extracted text data as the text materials extracted from the first object. In this implementation, for text materials, text views, picture views, file views, and web views that contain text data will be found from the first object and added to its view set, and the extraction will be performed based on the acquisition method of the first object, because the acquisition method of the first object will play a certain decisive role in the content included in the view set.

[0008] In combination with the first aspect, in certain implementations of the first aspect, when extracting text materials from the view set corresponding to the text materials, based on the acquisition method of the first object, extracting the text data in each view in the view set, and determining the extracted text data as the extracted text materials, it may include:

[0009] When the acquisition method of the first object is a long-press and drag operation on the first object, the view set corresponding to the text materials includes a text view and also includes a file view and / or a web view. The text data in the text view is extracted according to the pre-configured extraction rules in the configuration file corresponding to the first object, or, when the first object does not have a corresponding configuration file, the text data in the text view is extracted according to the custom extraction rules; or,

[0010] When the acquisition method of the first object is a three-finger swipe-down operation on the interface, the view set corresponding to the text materials includes a picture view of the interface screenshot of the first page, and the text is extracted from the picture view by means of optical character recognition (OCR).

[0011] In this implementation, for a long - press and drag operation, since the coordinates of the long - press can be determined, and the cases where text can be dragged out only include dragging after selecting a text paragraph in the display interfaces of the normal interface, file, and web page. Therefore, when it is a long - press and drag operation, the view set must include a text view, and may also include a file view or web view corresponding to the first interface. In this case, the text data in the text view can be extracted according to pre - configured rules or custom extraction rules. It should be understood that when a text paragraph is selected in the normal interface or an open file or an open web page, it is equivalent to the file view and web view containing a text view, and the first object is a drag operation after selection, which can only be a part of the first interface, rather than the entire interface. Therefore, in the long - press and drag operation, when extracting text materials, it is not necessary to consider the view corresponding to the entire interface, but only the text view in the selected content. For the three - finger swipe - down operation, since it is an operation on the entire page and no content is pre - selected, the first object may correspond to the entire page, or the entire web page or file corresponding to the page. Therefore, in the three - finger swipe - down operation, the text material corresponds to the entire image view of the interface screenshot.

[0012] Combined with the first aspect, in some implementations of the first aspect, when there is no configuration file corresponding to the first object, when extracting the text data in the text view according to the custom extraction rules, it may include:

[0013] Reading the text data of the text view from the accessibility node (AccessbilityNodeInfo) corresponding to the text view; or,

[0014] When the text data of the text view cannot be read from the AccessbilityNodeInfo, reading the text content from the text acquisition interface (getText) corresponding to the text view; or,

[0015] When the text data of the text view cannot be read from both the AccessbilityNodeInfo and getText, obtaining the text view identifier (ViewID) by directly calling the view, and then obtaining the text data of the text view according to the text ViewID; or,

[0016] When the text data cannot be read from the AccessbilityNodeInfo, getText, and ViewID, obtaining it from the view character acquisition interface (getViewString) of the text view by directly calling the view.

[0017] In this implementation, the custom extraction rules are some execution strategies sorted by execution priority. First, it preferentially reads from accessible nodes, which depends on the first application writing data to accessible nodes. Most current applications meet this condition. When encountering a few applications that do not meet this condition or when data cannot be read for other reasons, it then attempts other data reading methods in sequence. The above strategies range from simple to complex from front to back, and it starts by attempting the simplest reading method first.

[0018] Combined with the first aspect, in some implementations of the first aspect, when the material type set of the first object includes image materials, the view set corresponding to the image materials includes at least one of an image view, a web view, and a custom view; when extracting materials of each material type from the view set corresponding to the material type in the material type set of the first object, it may include: determining at least one view containing an image among the standard image view, the web view, and the custom view from the view set corresponding to the image materials according to the configuration corresponding to the view name or the application name; for the standard image view, obtaining the bitmap of the standard image view by reading the storage interface of the image, and determining the bitmap of the standard image view and the uniform resource identifier URI of the bitmap of the standard image view as the image material extracted from the first object; for the web view, obtaining the bitmap and URI of the image in the web view by injecting a JS script, and determining the bitmap of the web view and the URI of the bitmap of the web view as the image material extracted from the first object; for the custom view, copying the bitmap by traversing and finding the key attributes in the custom view, and determining the bitmap of the custom view and the URI of the bitmap of the custom view as the image material extracted from the first object; for the custom image view, when the bitmap cannot be found from the key attributes, drawing the bitmap of the custom image view, and determining the bitmap of the custom image view and the URI of the bitmap of the custom image view as the image material extracted from the first object.

[0019] In this implementation, different extraction strategies are determined and executed based on whether the view is an image view and whether the view is a standard image view. The standard image view is the ImageView. There may also be images in the WebView, so it is separately explained. For non-standard image views, that is, in custom views, if a Bitmap can be found, its Bitmap is read; if not, the Bitmap of this custom view is directly drawn. Figure 7 An example of the process of extracting an image is given. The above extraction strategy can extract image materials regardless of whether it is a standard image view, and with the fallback strategy of directly drawing the Bitmap, it can ensure that the images in all views containing images can be extracted.

[0020] Combined with the first aspect, in some implementations of the first aspect, when obtaining the Bitmap of the standard image view by reading the storage interface of the image, it may include: reading the Bitmap of the standard image view from the storage node of the standard image view; or, when the Bitmap of the standard image view cannot be read, drawing the Bitmap of the standard image view.

[0021] In this implementation, an execution strategy for finding the Bitmap of the standard image is given, giving priority to reading from the storage node, and directly drawing the Bitmap if not found. With the above strategy, even if the Bitmap cannot be read, there is a fallback strategy of directly drawing the Bitmap, which can ensure that the images in all views containing images can be extracted.

[0022] In combination with the first aspect, in certain implementations of the first aspect, when the material type set of the first object includes file materials, the view set corresponding to the file materials includes file views; for each material type in the material type set of the first object, when extracting the materials of that material type from the view set corresponding to that material type, it may include: when the first object is a file selected on the first page and the acquisition method of the first object is a long - press and drag operation on the first object, and the view set corresponding to the file materials only includes file views, directly extract the file information of the file view in the first object, check whether the file exists under the storage path of the file view, and when the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file materials extracted from the first object; or, when the first page is the display page for opening a file, the first object is the opened file, and the acquisition method of the first object is a three - finger swipe down operation on the first page, and the view set corresponding to the file materials includes the file view corresponding to the opened file, and also includes text views and / or picture views, search for the file information of the opened file in the pre - storage space, copy the file according to the file path indicated by the read file information, and determine the URI of the copied file and the read file information as the file materials extracted from the first object, or, when the file information of the opened file cannot be found in the pre - storage space within a preset time period after the three - finger swipe down operation, obtain the file information of the opened file through the fields of the activity, copy the file according to the file path indicated by the obtained file information, and determine the URI of the copied file and the obtained file information as the file materials extracted from the first object; the pre - storage space includes the file information of all detected opened files.

[0023] In this implementation, different extraction strategies for file views are given for the drag - and - drop scenario and the three - finger swipe down scenario respectively. For file collection, in the drag - and - drop scenario, it must be the selection and dragging of one or more files included in the interface. Whether these files are opened or not does not need to be concerned, so only operations such as extracting file information are required; for the three - finger swipe down scenario, there must be an interface (the first page) currently being displayed, and in this interface, the file being opened and displayed is shown. The three - finger swipe down is the three - finger swipe down on the opened file. At this time, the file view in the first object selected must be the file view corresponding to the opened file being displayed. At this time, it is necessary to find the file information of this file from the file information of all opened files stored in advance, and then trace back to this file for copying and URI extraction.

[0024] In combination with the first aspect, in some implementations of the first aspect, when the first object is a file selected on the first page and the acquisition method of the first object is a long - press and drag - and - drop operation on the first object, and when the view set corresponding to the file material only includes the file view, directly extract the file information of the file view in the first object, and check whether the file exists under the storage path of the file view. When the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object. This may include: determining whether the first application is connected to the file drag - and - drop framework; when the first application is connected to the file drag - and - drop framework, directly obtain the URI of the opened file; or, when the first application is not connected to the file drag - and - drop framework, scan and search for the file path of the opened file in the electronic device through the file name of the opened file, so as to obtain the URI of the opened file.

[0025] In this implementation, if this application (the first application) itself supports dragging of files, the URI of this file can be directly found. However, if this application does not support file dragging, it is still necessary to traverse and search for this file in the file system.

[0026] In combination with the first aspect, in some implementations of the first aspect, when the material type set of the first object includes web page materials, the view set corresponding to the web page materials includes webview; for each material type in the material type set of the first object, when extracting the material of this material type from the view set corresponding to this material type, this may include: extracting the title and body content of the webview in the first object by reading and parsing the readable readability file, and determining the extracted title and body content as the web page material extracted from the first object.

[0027] In this implementation, for the webview, the web page material in the form of an offline web page is extracted by parsing the content therein.

[0028] In combination with the first aspect, in some implementations of the first aspect, when the material type set of the first object includes video materials, the view set corresponding to the video materials includes webview; for each material type in the material type set of the first object, when extracting the material of this material type from the view set corresponding to this material type, this may include: determining the video playing area on the first page by reading the video configuration information written by the first application in the accessibility node, and performing at least one of the operations of screen recording, screenshotting, and adding tags to the currently playing video to obtain video excerpt content, and determining the obtained video excerpt content and the URI of the video excerpt content as the video material extracted from the first object.

[0029] In this implementation, for video materials, they must be partial or complete excerpts of the video played in weview or something similar to webview. Since the excerpting process exists, it is equivalent to the recording and extraction over a period of time. First, lock the video playback area (for example, when the video is not played full-screen on the web page), and then start operations such as screen recording and screenshotting to excerpt a segment or all of this video, thus obtaining the video excerpt material, which can also be called video material.

[0030] It should be noted that for videos, in addition to storing the excerpts of this video, the online link of this video can also be stored. In that case, the online link material is extracted by referring to the extraction method of the online link. This situation does not belong to the extraction of video materials. In addition, for videos in the case of mutual transmission of video files in chat applications, the videos exist in the form of files. The video file needs to be selected and dragged according to the drag-and-drop scenario. Therefore, the extraction of file materials can be referred to, but the extracted file material is a video.

[0031] Combined with the first aspect, in some implementations of the first aspect, when the material type set of the first object includes online link materials, the view set corresponding to the online link materials includes at least one of the online link view and the webview; for each material type in the material type set of the first object, the material of this material type is extracted from the view set corresponding to this material type, including: obtaining the online link material according to the uniform resource locator (URL) field carried in the activity module or by the way of getting the URL from the webview.

[0032] In this implementation, for online links, they can be found from the fields of the activity module or read from the webview. For some views, they are not webviews, but have properties similar to webviews and also have corresponding links, such as the online editing scenario of shared documents that often appears in work applications. Other situations are not listed here.

[0033] In one example, when obtaining online link materials according to the URL field carried in the activity or in the way of getting the URL from the webview, it may include: when the webview corresponding to the first page is a standard webview, directly use the way of getting the URL to obtain the online link of the standard webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is a custom webview, match the name of the webview from the configuration information of the first page to find the custom webview corresponding to the first page, and then use the way of getting the URL to obtain the online link of the custom webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is neither a standard webview nor a custom webview, obtain the online link of the webview from the URL field carried in the activity.

[0034] In this example, different ways of obtaining online links will be adopted according to whether the text view is a standard webview, a custom webview, or neither of them.

[0035] Combined with the first aspect, in some implementation manners of the first aspect, the above method further includes: sharing the first material to a second application. That is to say, the extracted material can be shared for application management, collection or use. The second application can be, for example, a collection space.

[0036] In a second aspect, there is provided an apparatus for extracting page materials, and the apparatus includes units composed of software and / or hardware for executing any one of the methods of the first aspect.

[0037] In a third aspect, there is provided an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it can implement any one of the methods of the first aspect.

[0038] In a fourth aspect, there is provided a chip, including a processor, and the processor is used to read and execute a computer program stored in a memory, and when the computer program is executed by the processor, it can implement any one of the methods of the first aspect.

[0039] Optionally, the chip further includes a memory, and the memory is electrically connected to the processor.

[0040] Optionally, the chip may further include a communication interface.

[0041] Fifth aspect, there is provided a computer-readable storage medium storing a computer program which, when executed by a processor, can implement any of the methods in the first aspect.

[0042] Sixth aspect, there is provided a computer program product comprising a computer program which, when executed by a processor, can implement any of the methods in the first aspect. Description of the Drawings

[0043] Figure 1 It is a schematic diagram of an interaction process of global collection in an embodiment of the present application.

[0044] Figure 2 and Figure 3 It is a schematic diagram of another interaction process of global collection in an embodiment of the present application.

[0045] Figure 4 It is a schematic flowchart of a method for extracting page materials in an embodiment of the present application.

[0046] Figure 5 It is a schematic diagram of a page material extraction framework in a long-press and drag operation scenario in an embodiment of the present application.

[0047] Figure 6 It is a schematic diagram of an execution framework for extracting text materials based on OCR in an embodiment of the present application.

[0048] Figure 7 It is a schematic flowchart of a method for extracting picture materials in an embodiment of the present application.

[0049] Figure 8 It is a schematic diagram of an execution framework for extracting file materials in a three-finger swipe-down scenario in an embodiment of the present application.

[0050] Figure 9 It is a schematic flowchart of a method for extracting file materials in a three-finger swipe-down scenario in an embodiment of the present application.

[0051] Figure 10 It is a schematic diagram of an execution framework for extracting web page materials in an embodiment of the present application.

[0052] Figure 11 It is a schematic diagram of an execution framework for extracting online links in an embodiment of the present application.

[0053] Figure 12 It is a schematic diagram of a device for extracting page materials in an embodiment of the present application.

[0054] Figure 13 It is a schematic diagram of the hardware structure of an electronic device in an embodiment of the present application. Specific Embodiments

[0055] The solutions of the embodiments of the present application will be introduced below with reference to the accompanying drawings.

[0056] Figure 1 It is a schematic diagram of an interaction process for global collection in an embodiment of the present application. As Figure 1 shown, assume that the user opens Application A on the electronic device and enters the interface shown in Interface 101, which includes some graphic and text data of a news item. Assume that the user selects a part of the content by pressing and sliding in Interface 101. As shown in Interface 102, a part of the text content and a picture content are selected. Assume that the user raises the hand after selecting the content and then long-presses on the selected content area. In response to this long-press operation, the electronic device displays Interface 103, in which the selected content is superimposed in a floating form on the interface. Assume that the user drags the floating selected content to the side of the electronic device while keeping the long-press without releasing, and the selected content can be added to the collection space application. As shown in Interface 104, the display interface of the collection space application is superimposed on the interface of Application A. Here, it is a long-strip interface, and it can be seen from the display interface of the collection space that the selected content has been added to the collection space, and the text and the picture are stored as a material respectively.

[0057] It should also be understood that Figure 1 taking Application A as an example, the page content in Application A is added to the collection space for collection. For other applications, the page content therein can also be added to the collection space, which will not be elaborated here. For example, the chat content in the chat application can be selected and dragged and then added to the collection space. Another example is that the mutually transmitted files in the chat application can be dragged to the collection space. Another example is that the attachments in the mail application can be dragged to the collection space. Another example is that the mail body content of the mail application can be selected and dragged to the collection space. Another example is that the pictures opened in the shopping application can be dragged to the collection space. Another example is that the pictures and videos in the gallery application can be dragged to the collection space. Other situations will not be listed one by one.

[0058] Figure 2 and Figure 3 It is a schematic diagram of another interaction process for global collection in an embodiment of the present application. As Figure 2As shown, assume that the user has opened Application A on the electronic device and entered the interface shown in Interface 104, which includes some graphic and text data of a news item. Assume that the user performs an operation of three-finger long press followed by three-finger swipe down in Interface 104. In response to this three-finger swipe down operation, the electronic device displays Interface 105, which includes, in addition to the interface content of Interface 104, three controls: web page collection, content extraction, and screenshot collection. It should be understood that in practice, the number, type, and form of the controls included in Interface 105 can be set as needed. It should also be understood that as the content included in Interface 104 varies, the number and type of controls in Interface 105 can also be different. For example, here Interface 104 is a web page, so Interface 105 includes web page collection and may also include link collection. Assume that what is displayed in Interface 104 is an open file. Taking a pdf document as an example, assume that an open pdf is displayed in Interface 104, then Interface 105 can include at least one of file collection, content extraction, and screenshot collection. Assume further that a picture is full-screen displayed in Interface 104, then Interface 105 can include picture collection (equivalent to file collection) and content extraction (specifically referring to the extraction of the content in this picture). Other cases will not be listed one by one.

[0059] Assume that the user clicks the content extraction control in Interface 105. In response to this click operation, the electronic device displays Interface 106. In the entire interface of Interface 106, the background color of the selectable content is different from that of the non-selectable content. Here, take the non-selectable background color as gray and the selectable background color as white as an example. Interface 105 also includes two controls: exit and collection. Assume that the user selects a part of the selectable content in Interface 105 by holding and swiping, as shown in Interface 107. It can be seen from Interface 107 that here, take selecting a piece of text content (Text Content #1) and two pictures (Picture Content #1 and Picture Content #2) as an example. The background color of the selected content is different from that of the unselected content and the non-selectable content. That is to say, in Interface 107, the selected content is marked by the background color. However, it should be understood that in practice, other methods can also be used to mark the selected content. For example, the color of the selected target content can be different from that of other selectable content and also different from that of other non-selectable content.

[0060] As Figure 3As shown, assume that the user raises their hand to end the selection operation after holding and sliding on interface 107, clicks on the favorite control. In response to this click operation, the electronic device adds the selected content to the favorite space. After the addition is complete, interface 108 is displayed. Interface 108 includes floating capsule (snackbar) #1. Floating capsule #1 includes a prompt for successful favoriting and a click-to-view control for indicating to view the favorite result. Assume that the user clicks on the "click to view" control in interface 108. In response to this click operation, the electronic device displays interface 109. As shown in interface 109, the display interface of the favorite space application is superimposed on the interface of application A. Here, the favorite space application is a long strip interface, and it can be seen from the display interface of the favorite space that the selected content has been added to the favorite space, and the text and pictures are stored as one material respectively. The selected piece of text and two pictures shown in interface 107 have all been favorited into the favorite space.

[0061] It should also be understood that Figure 2 and Figure 3 take application A as an example. After being triggered by a three-finger downward swipe, the page content in application A is added to the favorite space for favoriting. For other applications, the page content therein can also be added to the favorite space, which will not be elaborated here. For example, a three-finger downward swipe can be performed in the chat interface of a chat application, and after selecting content extraction and then selecting the target content, the target content can be favorited into the favorite space by clicking on the favorite control. Another example is that a three-finger downward swipe can be performed in the display interface of a mail application, and then select the material to add to the favorite space. The interaction process is similar to Figure 2 and Figure 3 and will not be elaborated here. Another example is that a three-finger downward swipe can be performed in a shopping application, and then select materials such as pictures to add to the favorite space. Another example is that a three-finger downward swipe can be performed in the display interface of a gallery application to add all the pictures and / or videos in the current interface to the favorite space. Other situations will not be listed one by one.

[0062] Assume that the user clicks on the web favorite space in interface 105. The entire web page corresponding to interface 104 can be favorited into the favorite space in the form of an offline web page such as a web page material. Assume that the user clicks on the screenshot favorite in interface 105. The screenshot of interface 104 can be favorited into the favorite space in the form of a picture material. Assume that the interface 105 also includes a link favorite control. When the user clicks on the link favorite control, the online link of the web page corresponding to interface 104 can also be favorited into the favorite space in the form of an online link material. Other situations will not be listed one by one.

[0063] Figures 1 - 3The materials to be extracted in the example include various forms of views such as web pages, online links, texts, and pictures, and it is also stated that in other collection scenarios, the collection of materials such as files and videos can also be included. Therefore, how to extract various types of materials becomes particularly important. To address this issue, the present application proposes a method capable of extracting different types of materials from a selected object.

[0064] Figure 4 is a schematic flowchart of a method for extracting page materials according to an embodiment of the present application. It can be seen that Figure 4 The method shown is Figures 1 - 3 an important part of the global collection in Figures 1 - 3 and helps to realize the functions shown in the whole Figure 4 The following will introduce each step shown in

[0065] S401. Obtain a first object from a first page.

[0066] The first page is the display interface of a first application running on an electronic device, and the first object contains a view of the material to be extracted. The first application can be, for example, Figures 1 - 3 the application A shown in

[0067] S402. Determine the material type set of the first object.

[0068] The material type set includes at least one material type among text, picture, file, web page, video, and online link.

[0069] The material type refers to the form in which the content is finally to be extracted, which is a different concept from the view type. The former is for the content, and the latter is for the carrier form of the content, or can be understood as the presentation form of the content.

[0070] Taking Figure 1 as an example, if the first object is a selected text paragraph and a picture, the material type corresponding to this text paragraph is text, and the material type corresponding to this picture is picture. Taking Figure 2For example, the first object is the entire interface 104, and the material types include text, pictures, web pages, and online links. The text material is, for example, the text content in the interface 104, the picture is, for example, the picture in the interface 104, the web page material is, for example, the interface 104 is a news web page, the web page material refers to this web page, and the online link material refers to the online link of this web page. Other situations will not be listed one by one.

[0071] S403. Determine the view set corresponding to each material in the first object in the material type set.

[0072] Each material corresponds to a view set, and each view set includes all views from which the material of the material type corresponding to the view set can be extracted.

[0073] It should be understood that the material type and the view type are two different concepts. The material type refers to the form of the content to be saved, such as being saved as text, picture, video, or file, etc. However, the view type refers to the carrier form when different contents are displayed on the electronic device. For example, an interface may include text, pictures, and videos, and the carriers of these text, pictures, and videos in the electronic device are called views.

[0074] Illustrated with an example, for a webview, it may include materials such as text, pictures, videos, and online links. For the picture material, the picture view will contain the picture material, the file view can also contain the material, and the webview can also contain the picture material. Therefore, in the embodiments of the present application, for the purpose of extracting page materials, considering that the types of carriers (that is, views) of different materials are also different, so first determine the material types included in the selected object (the first object), and then find all views in the first object that will contain this material for each material type.

[0075] S404. For each material type in the material type set of the first object, extract the material of this material type from the view set corresponding to this material type.

[0076] In one implementation, when the material type set of the first object includes text materials, the view set corresponding to the text materials includes at least one of a text view, a picture view, a file view, and a web view; when extracting materials of each material type in the material type set of the first object from the view set corresponding to the material type, it may include: when extracting text materials from the view set corresponding to the text materials, based on the acquisition method of the first object, extracting the text data in each view in the view set, and determining the extracted text data as the text materials extracted from the first object. In this implementation, for text materials, text views, picture views, file views, and web views that contain text data will be found in the first object and added to its view set. When extracting, it will be based on the acquisition method of the first object, because the acquisition method of the first object will play a certain decisive role in the content included in the view set.

[0077] In one implementation, when extracting text materials from the view set corresponding to the text materials, based on the acquisition method of the first object, extracting the text data in each view in the view set, and determining the extracted text data as the extracted text materials, it may include:

[0078] When the acquisition method of the first object is a long-press and drag operation on the first object, the view set corresponding to the text materials includes a text view and also includes a file view and / or a web view. Extract the text data in the text view according to the pre-configured extraction rules in the configuration file corresponding to the first object, or, when the first object does not have a corresponding configuration file, extract the text data in the text view according to the custom extraction rules; or,

[0079] When the acquisition method of the first object is a three-finger swipe-down operation on the interface, the view set corresponding to the text materials includes a picture view of the interface screenshot of the first page, and the text is extracted from the picture view by means of optical character recognition (OCR).

[0080] In this implementation, for the long - press and drag - and - drop operation, since the coordinates of the long - press can be determined, and the situation where text can be dragged out only includes dragging after selecting a text paragraph in the display interfaces of ordinary interfaces, files, and web pages. So when it is a long - press and drag - and - drop operation, the view set must include the text view, and may also include the file view or web view corresponding to the first interface. In this case, the text data in the text view can be extracted according to pre - configured rules or custom extraction rules. It should be understood that when a text paragraph is selected in an ordinary interface, an open file, or an open web page, it is equivalent to the file view and web view containing a text view, and the first object is a drag - and - drop operation after being selected, which can only be a part of the first interface, rather than the entire interface. Therefore, when extracting text materials during a long - press and drag - and - drop operation, there is no need to consider the view corresponding to the entire interface, but only the text view in the selected content. For the three - finger swipe - down operation, since it is an operation on the entire page and there is no pre - selected content, the first object may correspond to the entire page, or the entire web page or file corresponding to that page. So in the three - finger swipe - down operation, the text material corresponds to the entire picture view of the interface screenshot.

[0081] In another implementation, when there is no corresponding configuration file for the first object, when extracting the text data in the text view according to custom extraction rules, it may include:

[0082] Reading the text data of the text view from the accessibility node (AccessbilityNodeInfo) corresponding to the text view; or,

[0083] When the text data of the text view cannot be read from the AccessbilityNodeInfo, reading the text content from the text acquisition interface (getText) corresponding to the text view; or,

[0084] When the text data of the text view cannot be read from both the AccessbilityNodeInfo and getText, obtaining the text view identifier (ViewID) by directly calling the view, and then obtaining the text data of the text view according to the text ViewID; or,

[0085] When the text data cannot be read from the AccessbilityNodeInfo, getText, and ViewID, obtaining it from the view character acquisition interface (getViewString) of the text view by directly calling the view.

[0086] In this implementation, the custom extraction rules are some execution strategies sorted by execution priority. First, it preferentially reads from accessible nodes, which requires that the first application writes data to the accessible nodes. Most current applications meet this condition. When encountering a few applications that do not meet this condition or when data cannot be read for other reasons, it will then try other ways to read data in turn. The above strategies are from simple to complex from front to back, and it preferentially starts trying from the simplest reading method.

[0087] The following Figure 5 mentions an example of how to obtain text data in a drag scenario (long - press drag operation scenario). Figure 6 Gives an example of extracting text data through OCR recognition in a three - finger swipe - down scenario (three - finger swipe - down operation scenario).

[0088] In another implementation, when the material type set of the first object contains image materials, the view set corresponding to the image materials includes at least one of an image view, a web view, and a custom view;

[0089] When extracting materials of each material type in the material type set of the first object from the view set corresponding to the material type, it may include:

[0090] According to the configuration corresponding to the view name or application name, determine at least one view containing an image among the standard image view, web view, and custom view from the view set corresponding to the image material;

[0091] For the standard image view, obtain the bitmap of the standard image view by reading the storage interface of the image, and determine the unified resource identifier URI of the bitmap of the standard image view and the bitmap of the standard image view as the image material extracted from the first object;

[0092] For the web view, obtain the bitmap and URI of the image in the web view through JS script injection, and determine the bitmap of the web view and the URI of the bitmap of the web view as the image material extracted from the first object;

[0093] For the custom view, copy the bitmap by traversing and finding the key attributes in the custom view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the image material extracted from the first object;

[0094] For a custom image view, when the bitmap cannot be found from the key attributes, draw the bitmap of the custom image view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the image material extracted from the first object.

[0095] In this implementation method, different extraction strategies will be determined and executed based on whether the view is an image view and whether the view is a standard image view. The standard image view is the ImageView. There may also be images in the WebView, so it is explained separately. For non-standard image views, that is, in custom views, if the bitmap can be found, read its bitmap; if not, directly draw the bitmap of this custom view. Figure 7 An example of the process of extracting images is given. The above extraction strategy can extract image materials whether it is a standard image view or not, and with the strategy of directly drawing the bitmap as a fallback, it can ensure that the images in all views containing images can be extracted.

[0096] In an example, when obtaining the bitmap of a standard image view by reading the storage interface of the image, it may include: reading the bitmap of the standard image view from the storage node of the standard image view; or, when the bitmap of the standard image view cannot be read, drawing the bitmap of the standard image view.

[0097] In an example, the execution strategy for finding the bitmap of a standard image is given. First, read from the storage node, and if not found, directly draw the bitmap. With the above strategy, even if the bitmap cannot be read, there is still the strategy of directly drawing the bitmap as a fallback, which can ensure that the images in all views containing images can be extracted.

[0098] In one implementation method, when the material type set of the first object includes file materials, the view set corresponding to the file materials includes file views; when extracting the materials of each material type in the material type set of the first object from the view set corresponding to the material type, it may include:

[0099] When the first object is a file selected in the first page, and the acquisition method of the first object is a long-press and drag operation on the first object, and the view set corresponding to the file material only includes the file view, directly extract the file information of the file view in the first object, and check whether the file exists under the storage path of the file view. When the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object; or,

[0100] When the first page is the display page for opening a file, the first object is the opened file, and the acquisition method of the first object is a three-finger swipe down operation in the first page. The view set corresponding to the file material includes the file view corresponding to the opened file, and also includes a text view and / or a picture view. Search for the file information of the opened file in the pre-storage space, copy the file according to the file path indicated by the read file information, and determine the URI of the copied file and the read file information as the file material extracted from the first object. Or, when the file information of the opened file cannot be found in the pre-storage space within a preset time period after the three-finger swipe down operation, obtain the file information of the opened file through the fields of the activity, copy the file according to the file path indicated by the obtained file information, and determine the URI of the copied file and the obtained file information as the file material extracted from the first object; the pre-storage space includes the file information of all detected opened files.

[0101] In this implementation method, different extraction strategies for file views are given for the drag scenario and the three-finger swipe down scenario respectively. For file collection, in the drag scenario, it must be the selection and dragging of one or more files included in the interface. Whether these files are opened does not need to be concerned, so only operations such as extracting file information are required; for the three-finger swipe down scenario, there must be an interface (the first page) being currently displayed, and the file being opened and displayed is shown in this interface. The three-finger swipe down is a three-finger swipe down in the opened file. At this time, the file view in the first object selected must be the file view corresponding to the opened and displayed file. At this time, it is necessary to find the file information of this file from the file information of all opened files stored in advance, and then trace back to this file for copying and URI extraction.

[0102] In one example, when the first object is a file selected in the first page and the acquisition method of the first object is a long - press and drag operation on the first object, and when the view set corresponding to the file material only includes the file view, directly extract the file information of the file view in the first object, and check whether the file exists under the storage path of the file view. When the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object, which may include: judging whether the first application is connected to the file drag - and - drop framework; when the first application is connected to the file drag - and - drop framework, directly obtain the URI of the opened file; or, when the first application is not connected to the file drag - and - drop framework, scan and search for the file path of the opened file in the electronic device through the file name of the opened file, so as to obtain the URI of the opened file.

[0103] In this example, if this application (the first application) itself supports dragging of files, the URI of this file can be directly found. However, if this application does not support file dragging, it is still necessary to traverse and search for this file in the file system.

[0104] In another implementation manner, when the material type set of the first object contains web page materials, the view set corresponding to the web page materials includes webview; when extracting the materials of each material type in the material type set of the first object from the view set corresponding to the material type, it may include: extracting the title and body content of the webview in the first object by reading and parsing the readable readability file, and determining the extracted title and body content as the web page materials extracted from the first object.

[0105] In this implementation manner, for webview, extract the web page materials in the form of an offline web page by parsing the content therein.

[0106] In another implementation manner, when the material type set of the first object contains video materials, the view set corresponding to the video materials includes webview; when extracting the materials of each material type in the material type set of the first object from the view set corresponding to the material type, it may include: determining the video playing area in the first page by reading the video configuration information written by the first application in the accessibility node, and performing at least one of the operations of screen recording, screenshotting, and adding tags on the currently playing video to obtain video excerpt content, and determining the obtained video excerpt content and the URI of the video excerpt content as the video materials extracted from the first object.

[0107] In this implementation, for video materials, they must be partial or complete excerpts of videos played in a weview or something similar to a webview. Since the excerpting process takes time, it is equivalent to recording and extracting for a certain period. First, lock the video playback area (for example, when the video is not played full-screen on a web page), and then start operations such as screen recording and screenshotting to excerpt a segment or all of the video, thus obtaining video excerpt materials, which can also be called video materials.

[0108] It should be noted that for videos, in addition to storing the excerpts of the video, the online link of the video can also be stored. In that case, the online link materials are extracted according to the extraction method of the online link. This situation does not belong to the extraction of video materials. In addition, for videos in the case of mutual transmission of video files in a chat application, the videos exist in the form of files. The video file needs to be selected and dragged according to the drag-and-drop scenario. Therefore, the extraction of file materials can be referred to, except that the extracted file materials are a video.

[0109] It can be understood that videos in the materials in the collection space are mainly collected by dragging video files, while video excerpt materials can be excerpted from the video playback page. If the link of a video is stored in the collection space, the essence of this material is an online link, but it is just an online link associated with a video.

[0110] In another implementation, when the material type set of the first object includes online link materials, the view set corresponding to the online link materials includes at least one of the online link view and the webview; for each material type in the material type set of the first object, extract the materials of this material type from the view set corresponding to this material type, including: obtaining the online link materials according to the uniform resource locator (URL) field carried in the activity module or the way of reading the uniform resource locator (getURL) from the webview.

[0111] In this implementation, for online links, they can be found from the fields of the activity module or read from the webview. For some views, they are not webviews, but have properties similar to webviews and also have corresponding links, such as the online editing scenario of shared documents that often appears in work applications. Other situations are not listed here.

[0112] In one example, when obtaining online link materials according to the URL field carried in the activity or in the way of getting the URL from the webview, it may include: when the webview corresponding to the first page is a standard webview, directly use the way of getting the URL to obtain the online link of this standard webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is a custom webview, match the name of the webview from the configuration information of the first page, find the custom webview corresponding to the first page, and then use the way of getting the URL to obtain the online link of this custom webview, and the obtained online link is the online link material extracted from the first object; or, when the webview corresponding to the first page is neither a standard webview nor a custom webview, obtain the online link of this webview from the URL field carried in the activity.

[0113] In this example, different ways of obtaining online links will be adopted respectively according to whether the text view is a standard webview, a custom webview, or neither of them.

[0114] S405. After all view sets corresponding to the material type set of the first object are extracted, the first material corresponding to the first object is obtained.

[0115] That is to say, the first material is the material extracted from the first object.

[0116] It should be understood that although the first object may contain many types of materials, it is not necessarily to extract all materials, and a part of the materials can be selected for extraction based on the user's selection. Especially for the three-finger swipe-down scenario, there is no clear selected object before extracting the materials. The first object may correspond to part of the content, all of the content in the first interface, or may also correspond to the web page or file corresponding to the first interface.

[0117] In one implementation, the above method further includes: in response to the user's click operation on the extraction control, select the material type corresponding to the extraction control in the first object and add it to the above material type set.

[0118] In one implementation, the above method further includes: sharing the first material to a second application. That is to say, the extracted materials can be shared for application management, collection, or use. The second application can be, for example, a collection space.

[0119] Figure 4The method described above locks the views in the selected object that contain such materials according to the type of materials contained in the selected object, and then extracts the materials by classification, meeting the diversified material extraction requirements. The extracted page materials can be used for collection, document merging, or sharing to other applications or devices later.

[0120] It should also be understood that Figure 4 is a solution that first determines the type of materials to be extracted in the selected object, and then extracts different materials for different views. In addition to this solution, it is also possible to consider first determining all the view types contained in the selected object, and then for each view in each view type, extract the materials therein. However, this method still requires judging which materials can be extracted and which materials need to be extracted in each view. For text views and picture views that only contain one type of material, it is relatively simple. But for file views and web views, since they are often composite views, for example, a file view may further include a text view and a picture view, and may even include a video view. Therefore, when extracting materials from such views, different extraction methods for different materials in different views still need to be used respectively, and this method may result in redundant extractions. Moreover, since the extraction strategy determines the different extraction frameworks to be called based on the material type, and then different extraction strategies are adopted according to the differences in views under different extraction frameworks, in the above method, for the same view, different extraction frameworks need to be continuously switched to extract different materials until the extraction is completed. However, when extracting materials from all views where the material exists, the same extraction framework will be called, but different extraction strategies for different views may be adopted after calling the same framework.

[0121] Figure 5 It is a schematic diagram of a page material extraction framework in a long-press and drag operation scenario of an embodiment of the present application. Figure 5 It can be regarded as an example of an execution framework for extracting page materials triggered by a long-press and drag operation, or in a long-press and drag scenario (or drag scenario).

[0122] As Figure 5 shown, this execution framework includes an application process and a system service (systemserver) process. The system service process is mainly used to provide framework services such as background resources for the execution process in the application process. The application process includes an event intercepting process, a control classification process, an information extraction process, an information transfer process, and an information storage process.

[0123] The time-intercepting process mainly uses the activity-window to intercept drag events. In practice, after the event sensing module (eventHub) in the system service process receives the input events reported by the TP, it reports them to the input dispatching module (InputDispatcher) through the input reading module (InputReader). The Window Manager Service (WMS) manages the input dispatching module, and the InputDispatcher reports the input events to the application process. The activity-window in the application process intercepts the input events. Here, the input event is a long-press drag event. In the case of a long press, the coordinates of the long-press position (down(X, Y)) can be detected, and the activity-window also records the coordinates during the long press through the long-press coordinate recording module.

[0124] The control classification process mainly determines the classification of views and the types of materials that each view will contain. In the Android system, views include the corresponding views of materials such as text, pictures, files, and webviews. Text materials mainly exist in editable text (EditText), text views (TextView), and custom views. Picture materials mainly exist in standard picture views (ImageView) and non-standard imageviews, such as custom views and webviews. Combinations of views may exist in files. For example, it may be a combination of a text view (TextView) and a picture view (ImageView). When identifying the view type, the view type can be identified according to the file suffix and the control type. Webviews may also include text and / or picture materials, etc. Webviews can obtain the information therein through JS script injection. Other materials mainly refer to video materials, online link materials, etc. Video materials can mainly exist in webviews, file views, and custom views, and online link materials mainly exist in webviews and custom views.

[0125] The information extraction process includes text extraction, picture extraction, and file extraction. It should be understood that in the drag operation scenario, when dragging and collecting one or several pictures, these pictures exist in the form of files, so it is equivalent to the drag of a file. The drag of an entire video file is also equivalent to the drag of a file. For example, in a chat application, when sharing video files with each other, after downloading, long-pressing the video thumbnail and dragging it means dragging the entire video file in the form of a file.

[0126] When extracting text, it can be obtained through AccessbilityNodeInfo, getText, getViewID, or the view character storage interface (getViewString). When extracting images, the bitmap of the image can be obtained. For file extraction, in the case of having a file drag-and-drop framework, the file path can be directly obtained. If there is no file drag-and-drop framework, the uniform resource identifier (URI) of the file path can be obtained by scanning the file name to find the file path. For the extraction of other materials, the view set of the material can be locked according to the custom extraction policy, and then the material can be extracted from it.

[0127] When extracting text, if a third-party application writes text data into AccessbilityNodeInfo, the data can be directly retrieved from it. If not, it can be retrieved from other interfaces that store editable text data, such as getText here. If there is only text in the view state, the text data can be obtained by finding the identifier of the text view and then retrieving the text data based on the viewID. For text in the view state only, if the data still cannot be found based on the viewID, the text data can be retrieved from getViewString. In actual applications, for different applications, there will be some characteristics and configuration parameters of their own, that is, configuration files. Therefore, it can also be determined from the configuration files how to retrieve text data. For example, assuming that the configuration file can find the configuration description that text data will be written into AccessbilityNodeInfo, the text data can be directly retrieved from AccessbilityNodeInfo without having to try other methods. Another example, assuming that the configuration file can find the configuration description that text data will be written into getText but not into AccessbilityNodeInfo, the text data can be directly retrieved from getText without having to try other methods. Other situations will not be listed one by one.

[0128] When transmitting information, for text information, the text content is directly written into the clipboard base class (ClipData). For other materials such as images and files, their URIs are written into ClipData.

[0129] Through the binder communication channel, the extracted materials can be stored separately according to different storage policies corresponding to different materials. For example, text materials can be directly stored in the storage space of the favorites, and image materials can be stored in the storage space of the photo gallery, etc., which will not be elaborated here.

[0130] Private storage and public storage mean that some applications set storage limits, and the stored content is not shared with other applications, which belongs to the private storage of the application. Public storage is the storage space that all applications can use.

[0131] For file extraction, it should be understood that since in the drag-and-drop operation, it is already known that the dragged object is this file, and what is dragged is the mapping link of the file, rather than dragging in the display page of the file after it is opened, so it can be directly extracted based on the file path of this file, and there is no longer a need to monitor whether the file has been opened.

[0132] Figure 6 It is a schematic diagram of an execution framework for extracting text materials based on OCR in an embodiment of the present application. It should be noted that Figures 6 - 11 Regarding the extraction frameworks for different materials, for the same module, the function introductions of the same module can be mutually referenced for simplicity and will not be repeatedly introduced in each figure. Figure 6 It is an example of an execution framework for extracting text materials in the three-finger swipe-down scenario.

[0133] Such as Figure 6 As shown, the InputManager is used to obtain the three-finger press operation (for pulling up the global collection service) and the three-finger swipe-down operation (for triggering the global collection service). The HwFingersSnapshooter module in the input processor is used to recognize the gesture operations of three-finger press and three-finger swipe-down.

[0134] The input processor sends various operations obtained to the global collection service (global collection middleware platform). The global collection middleware platform includes the CollectionService module, SensorManager, ContentExtractActivity module, and TexExtractClient module.

[0135] Each extraction engine / module in the global collection middleware platform extracts the activity example under the trigger of the gesture module, and starts and schedules the corresponding processes through the activity manager service (AMS) to perform subsequent operations.

[0136] The global collection middleware platform can request a node (requestContentNode) from the AMS. The AMS, according to the request, feeds back the node to pass the activity instance through the ActivityThread in it, so as to send the screenshot of the interface during the three-finger swipe-down operation to the global collection middleware platform.

[0137] That is to say, after swiping down with three fingers, the global collection middle platform obtains the bitmap of the current page screenshot and the activity instance.

[0138] The global collection platform sends the initial data, that is, the data to be recognized, which is the bitmap of the current page screenshot and / or the activity instance, to the intelligent recognition platform AIPlugin, and receives the recognition result. The intelligent recognition platform includes VisionService, TextConfiguration, and OcrDetectEngine. VisionService is used to recognize the views in the interface, TextConfiguration is used to recognize text objects, and OcrDetectEngine is used to recognize the text in the picture, that is, to recognize the text in the picture (page screenshot) according to the glyph. Since the global collection middle platform receives the interface screenshot sent by AMS, the text in the interface screenshot and the text in the picture in the interface screenshot can both be recognized by OCR.

[0139] The global collection middle platform requests OCR recognition from AIPlugin, then sends the interface screenshot to AIPlugin and receives the text recognized by OCR.

[0140] It should also be understood that for the content extraction of an interface screenshot, the extraction of text materials therein is mainly considered here. However, in fact, the interface may also include pictures. At this time, the picture extraction engine in the global collection needs to be used to extract the pictures contained in the interface screenshot, and then the extracted pictures and the above-mentioned text data recognized by OCR are arranged in coordinate order to obtain the content extraction result of the entire interface. The extraction result may only include text or may also include pictures.

[0141] Combined Figure 2 and Figure 3When the user performs a three - finger press operation in the interface 104, this press operation will be captured by the input processor and transmitted to the global collection service middleware. Then, the global collection service middleware launches the global collection service. When the user performs a three - finger swipe - down operation based on the three - finger press operation, this three - finger swipe - down operation will be captured by the input processor and transmitted to the global collection service middleware. Then, the collection service module of the global collection service middleware collects the content in the interface, requests a node from the AMS, and displays the interface 105 in the form of an activity instance. When the user clicks on content extraction in the interface 105, the input processor captures this click operation and transmits it to the global collection service middleware. The text extraction module in the global service middleware interacts with the intelligent recognition platform to perform content extraction on the interface screenshot, thereby determining the display result of the text shown in the interface 106.

[0142] It should also be noted that there are essential differences between view recognition + OCR recognition and the recognition of traditional image edge recognition algorithms. The recognition of traditional image edge recognition algorithms (such as the ROI detection algorithm for recognition) may be able to identify various elements in a picture through edges, such as people, objects, and even text in it, but it cannot recognize the picture. Especially in the solution of the embodiments of the present application, OCR recognition and picture view extraction are performed on the basis of the interface screenshot after the interface is captured. However, if an image edge detection algorithm is used, it cannot lock which small pictures and which texts are included in a large picture, and can only recognize all possible targets such as people and objects in the entire large picture.

[0143] Figure 7 It is a schematic flowchart of extracting picture materials in the embodiments of the present application. The following introduces Figure 7 each step shown below.

[0144] S701. Initiate picture content extraction through Activity.

[0145] S702. In the Activity, screen out all views that need to extract pictures.

[0146] Before extracting picture materials, it is necessary to first find the views containing pictures, mainly involving standard imageview, webview, and Huawei custom views.

[0147] S703. The Activity sends all views that need to extract pictures to the picture collection module (ImageCollect).

[0148] S704. In ImageCollect, according to the configuration corresponding to the view name or application name, determine the extraction strategy that needs to be adopted when extracting pictures for each view.

[0149] That is to say, there are differences in the ways that different types of views can adopt when extracting the pictures therein.

[0150] S705. In ImageCollect, for a standard picture view, read the bitmap from its storage node.

[0151] S706. In ImageCollect, when the bitmap cannot be read, draw the standard picture view into a bitmap.

[0152] S707. In ImageCollect, when the view is not a standard image view, draw the view into a bitmap.

[0153] The non - standard image view refers to a custom image view.

[0154] S708. In ImageCollect, when the view is a webview, obtain the bitmap by means of JS script injection.

[0155] The above - mentioned extraction process is to first determine whether the view is a standard picture view, a webview or a custom view, and then extract according to the extraction methods of different types of picture views.

[0156] S709. ImageCollect writes the bitmap to the file system.

[0157] S710. In the file system, write a file, and obtain the URI after the storage is completed.

[0158] Figure 8 It is a schematic diagram of an execution framework for extracting file materials in the scenario of three - finger swipe down in an embodiment of the present application. As Figure 8As shown, after the input processor receives a three-finger downward sliding operation, it sends it to the application process according to the whitelist indication. The Activity module in the application process includes an Intent module, a FeedData module, and a FileName module to provide file-related data. The path acquisition module in the application process is used to obtain the storage path of the file, and the file transfer module is used to copy the file. When the file is already in the public storage space, it can be not copied. When the file is in the private storage space, to prevent subsequent restrictions by the application and inability to access the file again, the file is copied to the public storage space or the storage space corresponding to the global favorite function. The file management service is used to provide services related to file material extraction for the global favorite. The file management service includes a FileManager for managing files and a FiledInfoProvider for extracting file information. Here, the file management service can perform real-time monitoring (detection) of files, that is, detect the opened files, and then when the file information needs to be viewed in the global favorite, send the monitored file information to the global favorite middle platform. The Istalled module includes the command-line tool Inotify and the FileExecutor, which are used to provide services for opening, viewing, and editing files, and here are also used to find the hidden public storage path from the file system and send the hidden public storage path to the CollectionProvider module in the global favorite middle platform.

[0159] Triggered by the three-finger downward sliding operation, the global favorite middle platform obtains the file information of the file that is in the open state and corresponds to the file at the time when the three-finger downward sliding operation is executed from the file management service. For example, it may be that a three-finger downward sliding operation is performed on the display interface after file A is opened. Through the CollectionService of the global favorite middle platform, the file information is sent to the file system to find the corresponding storage path of the file system, and the URI of the file is stored in the database (data storage unit) under CollectionProvider. The global favorite middle platform also provides a whitelist for the input processor to guide whether file material extraction can be performed.

[0160] That is to say, there are three ways to obtain file materials: find from the monitored file information, directly obtain through the path acquisition module in the application process, or obtain after assembly.

[0161] Figure 9 It is a schematic flowchart of extracting file materials in a three-finger downward sliding scenario in an embodiment of the present application.

[0162] S901. The file monitoring module initializes and starts the file monitoring service.

[0163] S902. The file monitoring module starts to execute the file monitoring service and monitors in real time whether there is a file being opened.

[0164] S903. The user opens any file in the application interface.

[0165] S904. The application interface reports the file opening event to the file monitoring module.

[0166] S905. The file monitoring module obtains the file information of the opened file.

[0167] S906. The file monitoring module reports the obtained file information to the document extraction framework.

[0168] S907. The document extraction framework saves the file information reported by the file monitoring module.

[0169] In one implementation, the storage here can be to store only the information of the file that is being opened and displayed, that is, the open state is presented to the user on the application interface. Then whenever the user switches the application interface and this file is no longer displayed on the interface, it is considered that this file is no longer in the open and displayed state and is deleted from the document extraction framework.

[0170] In another implementation, the storage here can also be the file information of all files in the open state on the electronic device. Here, "open" means being opened on the electronic device, or understood as not being in the closed state. For example, if two files are opened at the same time, one is displayed in the foreground and the other is suspended in the background, both are open files. This method is to prevent the situation that after the user temporarily views other files and then comes back to view the current file, if following the previous real-time update method, the current file will first become not in the open state and then become in the open state again, resulting in redundant operations of deleting information and re-storing information.

[0171] In another implementation, only the file information of the files opened within a preset time period from the current moment is stored. For example, the file information of the files opened within 10 minutes can be stored. This method can prevent the file information of the files that the user has not closed all the time from occupying the storage space all the time, and prevent too much file information from being stored, which makes it necessary to search through a lot of file information when searching.

[0172] In yet another implementation, only the file information within a preset quantity threshold range is stored. For example, only the file information of less than or equal to 2 files can be stored. This implementation restricts the operation of storing file information in terms of the storage quantity, making the stored file information within a certain range. The preset quantity threshold range can be set as needed. For example, for a scenario that does not support split-screen operation and cannot display multiple files simultaneously, a relatively small value can be set; for a scenario that supports multi-screen display of multiple files, a relatively large value can be set.

[0173] The above various implementations can be reasonably combined.

[0174] S901 - S907 are operations that can be continuously executed, regardless of whether the three-finger swipe-down operation is executed. However, to prevent the file monitoring from continuously running in the background, consuming resources and power, a trigger condition can be added before step S901 to make the monitoring execute only under certain conditions.

[0175] In one implementation, the file monitoring is started in response to the startup operation of an application with file viewing and / or editing functions. That is, the file monitoring is only executed when there is a possibility of opening a file.

[0176] S908: The user performs a three-finger swipe-down operation on file A opened in the application interface.

[0177] S909: The application interface reports a file collection task, that is, a file material extraction task, to the document extraction framework. The document extraction framework obtains the file information of file A from all the stored file information according to the collection task. When the file information of file A cannot be obtained, the file information of file A is obtained from the activity field.

[0178] That is to say, in the three-finger swipe-down operation, it must be an operation performed on the display interface of the opened file. Therefore, it is preferred to search from the stored file information first, and if it cannot be found, search from the activity.

[0179] It should also be understood that since any file is involved in step S903, it may not necessarily be file A in S908. Also, when performing the three-finger swipe-down in S908, it indicates that file A has already executed step S903, but it is unknown whether steps S903 - S907 have been completed. Therefore, when step S910 starts to execute, the file information of file A may not have been stored yet because the corresponding step S907 of file A has not been completed, and thus the file information of file A may not be found in step S910.

[0180] Figure 10 It is a schematic diagram of an execution framework for extracting web page materials in an embodiment of the present application. As Figure 10As shown, the three-finger down swipe operation obtained from the InputManager is sent to the global collection service. The CollectionService in the global collection service distributes the three-finger down swipe event and sends a web page collection request to the CollectionService in the note application through binder communication. After receiving the web page collection request, the CollectionService in the note application needs to store the web page content. The SensorManager obtains the URL of the web page and copies the html file (mht file) of the web page. The webview module in the note application loads the web page content, performs the parsing of the web page content, and calls the title and body extraction algorithms in the extraction module to parse out the title and body content of the web page, and then refreshes the page to the webview module. The webview module sends the content of the offline web page and the unique identifier UUID of the offline web page in the note application to the CollectionProvider in the global collection service. The CollectionProvider receives the web page materials (including the extracted title, body content, and UUID), and then stores the data to be stored in the data storage unit Collections.db in the global collection service according to the storage mechanism of the web page materials.

[0181] Figure 11 It is a schematic diagram of an execution framework for extracting online links in an embodiment of the present application. It should be understood that Figure 11 the online links in it include web page links and video links. The three-finger down swipe operation obtained from the InputManager is sent to the video extraction application or the application process. When video extraction is required, the video extraction module in the video extraction application performs video extraction according to Figure 11 similar steps as shown. The video extraction module includes a floating tool window / bar for obtaining user interaction operations, and also includes scene recognition, which can perform operations such as video extraction, screenshot, and adding sticky notes on the video. On the application process side, the web page link can be directly obtained through the connection acquisition module, or first assembled and then obtained, and then the web page link is sent to the video link in the video extraction application. The video link module includes a configuration management and a category recognition module. The activity in the application process includes Intent, video identifier (MediaID), and a play module (playProxy).

[0182] The video extraction application sends the extracted video to the global collection middle platform, and the global collection middle platform obtains the video link or the web page link file system and the URI for obtaining the content of the video extraction.

[0183] The AccessibilityService, MediaCodecServer in the framework layer (FWK), and the WindowManager are used to provide background support services for the video extraction application. The global collection middle platform sends the whitelist to the InputManager.

[0184] It should be understood that for a web page, it may include videos. A user may want to save the link of the entire web page, or may only want to save the specific link of a certain video, or may only want to extract a part of a certain video. Therefore, Figure 11 there will be cases where both the video link and the web page link shown exist. For the extraction of online links that have no video extraction requirements at all, the part Figure 11 related to video extraction in it can be removed.

[0185] In one implementation, when the current page is a standard webview, directly obtain the online link of the webview according to the GetURL method of the webview; or,

[0186] when the current page is not a standard webview, for example, it may be a custom webview. The name of the webview can be matched from the configuration information of the current page, and then this webview can be found. After that, the online link of the webview can be obtained according to the GetURL method; or,

[0187] when the current page is neither a standard webview nor a custom webview, the online link can be read from the fields of the activity.

[0188] The basis for the above judgment (whether it is a webview or some other view) can be found from the whitelist configuration information.

[0189] The above mainly introduced the method of the embodiments of the present application in combination with the drawings. It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence, these steps are not necessarily executed in the order shown in the figures. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps. Next, the device of the embodiments of the present application will be introduced in combination with the drawings.

[0190] Figure 12 is a schematic diagram of an apparatus for extracting page materials according to an embodiment of the present application. As Figure 12 shown, the apparatus 2000 includes an acquisition unit 2001 and a processing unit 2002. The apparatus 2000 may be integrated in electronic devices such as mobile phones, tablet computers, touch-screen laptop computers, etc.

[0191] The apparatus 2000 is capable of performing any of the above methods. For example, the acquisition unit 2001 may be used to perform step S401, and the processing unit 2002 is used to perform steps S402 - S405. The apparatus 2000 may also be used to perform Figures 1 - 3 the interaction process shown. The apparatus 2000 may also be used to perform Figures 5 - 11 the execution process or steps shown.

[0192] In one implementation, the apparatus 2000 may further include a storage unit for storing relevant data. The storage unit may be integrated in any of the above units or may be a unit independent of all the above units.

[0193] Figure 13 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. As Figure 13 shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, antenna 1, antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, a key 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Among them, the sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0194] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 200. In some other embodiments of the present application, the electronic device 200 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0195] Exemplarily, Figure 13 the illustrated processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0196] Among them, the controller may be the nerve center and command center of the electronic device 200. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0197] A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can save the instructions or data that the processor 210 has just used or recycled. If the processor 210 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0198] The electronic device 200 realizes the display function through the GPU, the display screen 294, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change the display information.

[0199] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0200] The electronic device 200 can implement the shooting function through the ISP, camera 293, video codec, GPU, display screen 294, application processor, etc.

[0201] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 200 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0202] In the embodiments of the present application, OCR recognition can be performed by means of the NPU or GPU.

[0203] The internal memory 221 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.). The data storage area can store the data created during the use of the electronic device 200 (such as audio data, phone book, etc.). In addition, the internal memory 221 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0204] In the embodiments of the present application, the extracted materials can be stored in the internal memory or external memory, or the URI or URL of the materials can be stored in the internal memory, and the source files can be stored in the external memory, etc., which will not be listed one by one.

[0205] The pressure sensor 280A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 280A may be disposed on the display screen 294. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 280A, the capacitance between the electrodes changes. The electronic device 200 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 294, the electronic device 200 detects the intensity of the touch operation according to the pressure sensor 280A. The electronic device 200 can also calculate the position of the touch according to the detection signal of the pressure sensor 280A. In some embodiments, touch operations with the same touch position but different touch operation intensities may correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0206] The touch sensor 280K, also called the "touch panel". The touch sensor 280K may be disposed on the display screen 294. The touch sensor 280K and the display screen 294 form a touch screen, also called the "touch display screen". The touch sensor 280K is used to detect touch operations acting on it or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In some other embodiments, the touch sensor 280K may also be disposed on the surface of the electronic device 200, at a different position from that of the display screen 294.

[0207] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of the present application, their specific functions and the technical effects brought thereby can be specifically referred to in the method embodiment part, and will not be elaborated here.

[0208] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0209] An embodiment of this application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the electronic device can implement the steps in any of the above methods.

[0210] An embodiment of this application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by an electronic device, it can implement the steps in each of the above method embodiments.

[0211] An embodiment of this application provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, it can implement the steps in each of the above method embodiments.

[0212] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0213] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0215] In the embodiments provided in this application, it should be understood that the disclosed device / equipment and method can be implemented in other ways. For example, the device / equipment embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0216] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] It should be understood that when used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0218] It should also be understood that the term "and / or" as used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0219] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0220] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0221] The reference to "one embodiment" or "some embodiments" etc. described in the specification of this application means that a specific feature, structure or characteristic described in combination with this embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0222] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included within the protection scope of the present application.

Claims

1. A method for extracting page materials, characterized in that, Including: Obtain a first object from a first page, where the first page is a display interface of a first application running on an electronic device, and the first object contains a view that is the view of the material to be extracted; Determine the material type set of the first object, where the material type set includes at least one of the material types of text, picture, file, web page, video, and online link; Determine the view set corresponding to each material in the material type set in the first object. Each material corresponds to a view set, and each view set includes all views that can extract the material of the material type corresponding to the view set; For each material type in the material type set of the first object, extract the material of the material type from the view set corresponding to the material type; After all the view sets corresponding to the material type set of the first object are extracted, obtain the first material corresponding to the first object.

2. The method according to claim 1, characterized in that, When the material type set of the first object includes text material, the view set corresponding to the text material includes at least one of text view, picture view, file view, and web view; for each material type in the material type set of the first object, extracting the material of the material type from the view set corresponding to the material type includes: When extracting text material from the view set corresponding to the text material, based on the acquisition method of the first object, extract the text data in each view in the view set, and determine the extracted text data as the text material extracted from the first object.

3. The method according to claim 2, wherein The step of, when extracting text material from the view set corresponding to the text material, based on the acquisition method of the first object, extracting the text data in each view in the view set and determining the extracted text data as the extracted text material includes: When the acquisition method of the first object is a long-press and drag operation on the first object, the view set corresponding to the text material includes a text view and also includes a file view or a web view, and extract the text data in the text view according to the pre-configured extraction rules in the configuration file corresponding to the first object, or, when the first object does not have a corresponding configuration file, extract the text data in the text view according to the custom extraction rules; or, When the acquisition method of the first object is a three-finger swipe-down operation on the interface, the view set corresponding to the text material includes a picture view of the screenshot of the first page, and extract text from each view in the view set corresponding to the text material by means of optical character recognition (OCR).

4. The method according to claim 3, wherein The step of, when the first object does not have a corresponding configuration file, extracting the text data in the text view according to the custom extraction rules includes: Read the text data of the text view from the AccessibilityNodeInfo corresponding to the text view; or, When the text data of the text view cannot be read from the AccessibilityNodeInfo, read the text content from the getText interface corresponding to the text view; or, When the text data of the text view cannot be read from both the AccessibilityNodeInfo and getText, obtain the text view identifier ViewID by directly calling the view, and then obtain the text data of the text view according to the ViewID; or, When the text data cannot be read from the AccessibilityNodeInfo, getText, and ViewID, obtain it from the getViewString interface of the view characters of the text view by directly calling the view.

5. The method according to claim 1, wherein When the material type set of the first object includes picture materials, the view set corresponding to the picture materials includes at least one of a picture view, a web view, and a custom view; For each material type in the material type set of the first object, extract the material of this material type from the view set corresponding to this material type, including: According to the configuration corresponding to the view name or application name, determine at least one view containing a picture among the standard picture view, web view, and custom view from the view set corresponding to the picture material; For the standard picture view, obtain the bitmap of the standard picture view by reading the storage interface of the picture, and determine the bitmap of the standard picture view and the uniform resource identifier URI of the bitmap of the standard picture view as the picture material extracted from the first object; For the web view, obtain the bitmap and URI of the picture in the web view by injecting a JS script, and determine the bitmap of the web view and the URI of the bitmap of the web view as the picture material extracted from the first object; For the custom view, copy the bitmap by traversing and finding the key attributes in the custom view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the picture material extracted from the first object; For the custom picture view, when the bitmap cannot be found from the key attributes, draw the bitmap of the custom picture view, and determine the bitmap of the custom view and the URI of the bitmap of the custom view as the picture material extracted from the first object.

6. The method according to claim 5, characterized in that, The step of obtaining the bitmap of the standard picture view by reading the storage interface of the picture for the standard picture view includes: Read the bitmap of the standard picture view from the storage node of the standard picture view; or, When the bitmap of the standard picture view cannot be read, draw the bitmap of the standard picture view.

7. The method according to claim 1, wherein When the material type set of the first object includes file materials, the view set corresponding to the file materials includes file views; for each material type in the material type set of the first object, extracting the material of the material type from the view set corresponding to the material type includes: When the first object is a file selected on the first page, the acquisition method of the first object is a long - press and drag operation on the first object, and the view set corresponding to the file material only includes file views, directly extract the file information of the file view in the first object, check whether the file exists under the storage path of the file view, when the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object; or, When the first page is the display page for opening a file, the first object is the opened file, the acquisition method of the first object is a three - finger swipe - down operation on the first page, the view set corresponding to the file material includes the file view corresponding to the opened file, and also includes text views and / or picture views, search for the file information of the opened file in the pre - storage space, copy the file according to the file path indicated by the read file information, and determine the URI of the copied file and the read file information as the file material extracted from the first object, or, when the file information of the opened file cannot be found in the pre - storage space within a preset time period after the three - finger swipe - down operation, obtain the file information of the opened file through the fields of the activity, copy the file according to the file path indicated by the obtained file information, and determine the URI of the copied file and the obtained file information as the file material extracted from the first object; the pre - storage space includes the file information of all detected opened files.

8. The method according to claim 7, characterized in that, The description of "When the first object is a file selected on the first page, the acquisition method of the first object is a long - press and drag operation on the first object, and the view set corresponding to the file material only includes file views, directly extract the file information of the file view in the first object, check whether the file exists under the storage path of the file view, when the file exists under the storage path of the file view, copy the file, and determine the URI of the copied file and the extracted file information as the file material extracted from the first object" includes: Judge whether the first application is connected to the file drag - and - drop framework; When the first application is connected to the file drag - and - drop framework, directly obtain the URI of the opened file; or, When the first application is not connected to the file drag-and-drop framework, scan and search for the file path of the opened file in the electronic device by the file name of the opened file, so as to obtain the URI of the opened file.

9. The method according to claim 1, characterized in that When the material type set of the first object includes web materials, the view set corresponding to the web materials includes webview; for each material type in the material type set of the first object, extracting the materials of this material type from the view set corresponding to this material type includes: By reading and parsing the readability file, extracting the title and body content of the webview in the first object, and determining the extracted title and body content as the web materials extracted from the first object.

10. The method according to claim 1, wherein When the material type set of the first object includes video materials, the view set corresponding to the video materials includes webview; for each material type in the material type set of the first object, extracting the materials of this material type from the view set corresponding to this material type includes: By reading the video configuration information written by the first application in the accessibility node, determining the video playback area in the first page, and performing at least one of the operations of screen recording, screenshotting, and adding tags to the currently playing video to obtain video excerpt content, and determining the obtained video excerpt content and the URI of the video excerpt content as the video materials extracted from the first object.

11. The method according to claim 1, wherein When the material type set of the first object includes online link materials, the view set corresponding to the online link materials includes at least one of an online link view and a webview; for each material type in the material type set of the first object, extracting the materials of this material type from the view set corresponding to this material type includes: Obtaining the online link materials according to the uniform resource locator URL field carried in the activity or by reading the uniform resource locator getURL from the webview.

12. The method according to claim 11, wherein The obtaining the online link materials according to the uniform resource locator URL field carried in the activity or by reading the uniform resource locator getURL from the webview includes: When the webview corresponding to the first page is a standard webview, directly using the getURL method to obtain the online link of this standard webview, and the obtained online link is the online link material extracted from the first object; or, When the webview corresponding to the first page is a custom webview, matching the name of the webview from the configuration information of the first page, finding the custom webview corresponding to the first page, and then using the getURL method to obtain the online link of this custom webview, and the obtained online link is the online link material extracted from the first object; or, When the webview corresponding to the first page is neither a standard webview nor a custom webview, obtain the online link of the webview from the URL field carried in the activity.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Share the first material to a second application.

14. An electronic device, comprising a memory, one or more processors, and a computer program stored in the memory and executable on the processors, characterized in that, When the one or more processors execute the computer program, the electronic device implements the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by an electronic device, the method according to any one of claims 1 to 13 is implemented.

Images