Text content acquisition method and electronic equipment

By obtaining the root node view and control collection of the interface, combining text acquisition methods and optical character recognition, the problem that electronic devices cannot obtain all text content in the interface is solved, improving the user experience and providing convenient text operation functions.

CN119065757APending Publication Date: 2024-12-03HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310642483.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Electronic devices cannot obtain all the text content displayed in the interface, resulting in poor user experience and cannot meet the needs of copying and pasting text content.

Method used

By receiving user operations, obtaining the root node view of the interface, traversing the control collection, using methods such as getText(), getTooltipText(), getContentDescription() or JAVA reflection mechanism, obtaining the text content of the control, and performing optical character recognition when necessary. Make sure to get all the text content displayed in the interface.

Benefits of technology

It realizes that electronic devices can obtain all the text content displayed in the interface, improve user experience, provide convenient access to text content, understand user behavior habits and make relevant recommendations to prevent forgetting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065757A_ABST
    Figure CN119065757A_ABST
Patent Text Reader

Abstract

The invention provides a text content acquisition method and electronic equipment. In the method, the electronic equipment can obtain text content through a related method function; the electronic equipment can also use a JAVA reflection mechanism to obtain the text content; if the control containing the text content is not in the level touched by the user, the electronic equipment can also acquire the text content in the control by traversing a parent class control; the electronic equipment can also perform character extraction on the picture to obtain text content. According to the technical method, the problem that the electronic equipment cannot acquire certain text contents can be solved. In this way, the electronic equipment can obtain all the displayed text content in the interface, the experience feeling of a user is improved, and guarantee and convenience are provided for the user to obtain the text content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method for acquiring text content and an electronic device. Background Art

[0002] With the development and progress of electronic devices, electronic devices can respond to user operations (such as long press operations) on the specified area to copy or search for text content in the specified area. However, in some cases, the text content is not stored in the control level touched by the user, which will result in not all text content displayed in the interface being acquired by the electronic device, and then the text content can be copied and pasted. How to meet the user's needs for all text content displayed in the interface (such as copy and paste, etc.) is a problem that needs to be solved in this field. Summary of the invention

[0003] The present application provides a method and electronic device for acquiring text content, which can solve the problem that the electronic device cannot acquire certain text content. In this way, the electronic device can acquire all the text content displayed in the interface, improve the user experience, and provide protection and convenience for the user to acquire text content.

[0004] In a first aspect, the present application provides a method for obtaining text content, the method comprising: receiving a first operation for a first area in a first interface. Obtaining a root node view of the first interface. Traversing the root node view of the first interface to determine a first control set, the first control set including controls in the first area. Obtaining first text content, the first text content including text content of controls in the first control set. Recommending a first function, the first function being related to the first text content.

[0005] After implementing the method provided by the present application, the problem that the electronic device cannot obtain certain text content is solved. In this way, the electronic device can obtain all the text content displayed in the interface, improve the user experience, and provide protection and convenience for users to obtain text content. In addition, by obtaining text content, the user's behavior habits can be understood, so as to recommend relevant content to the user, prompt the user according to the user's behavior habits, and prevent the user from forgetting.

[0006] In combination with the first aspect, in some implementations, after obtaining the first text content, the method further includes: displaying a selection box, the range of which includes the first area. Receiving a fourth operation for the selection box, the fourth operation is used to change the position and / or size of the selection box. Traversing the root node view of the first interface, determining a third control set, the third control set including the controls in the selection box. Obtaining a third text content, the third text content including the text content of the controls in the third control set.

[0007] This is because after the user performs the first operation, the text content in the position touched by the first operation may not be what the user expects to obtain. The electronic device can also determine the area where the text content is to be obtained according to the user's needs.

[0008] In combination with the first aspect, in some implementations, after receiving the first operation for the first area in the first interface, the method further includes: obtaining the first coordinates of the first operation in the first interface. Determining the first control set specifically includes: obtaining the coordinate ranges of multiple controls in the root node view. The controls whose coordinate ranges of the controls in the root node view include the first coordinates are determined as controls in the first control set. In this way, the first control set within the first range can be determined so that text content can be subsequently obtained.

[0009] In conjunction with the first aspect, in some implementations, obtaining the coordinate range of multiple controls in the root node view specifically includes: obtaining position information of multiple controls in the root node view. Obtaining the coordinate range according to the position information of the multiple controls. In some implementations, the electronic device can obtain the left, top, right, and bottom attributes of the control and then determine the coordinate range of the control.

[0010] In conjunction with the first aspect, in some implementations, obtaining the first text content specifically includes:

[0011] For the second control, use the getText() method function or the getTooltipText() method function or the getContentDescription() method function to obtain the text content of the second control. Or obtain the class of the second control; obtain the declaration field attribute according to the class; traverse the declaration field attribute, obtain the field whose declaration field attribute is a string attribute, and determine the field as the text content of the second control. Or use JAVA's reflection mechanism to obtain the return value, and determine the return value as the text content of the second control. Or when the second control is of TextureView type, obtain the image information of the second control. Perform optical character recognition (OCR) processing on the image information to obtain the text content of the second control. Among them, the second control is any one of the controls in the first control set. In this way, the text content in the first area can be obtained by the above method.

[0012] In a second aspect, the present application provides a method for obtaining text content, characterized in that the method comprises: receiving a first operation on a first control. Obtaining a parent control of the first control. Obtaining a second control set, the second control set including child controls of the parent control. Obtaining a first text content, the first text content including the text content of controls within the scope covered by the first control in the second control set. Recommending a first function based on the text content, the first function being related to the first text content.

[0013] After implementing the method provided by the present application, the problem that the electronic device cannot obtain certain text content is solved. In this way, the electronic device can obtain all the text content displayed in the interface, improve the user's experience, and provide protection and convenience for users to obtain text content. In addition, by obtaining the text content in the control, the user's behavior habits can be understood, so as to recommend relevant content to the user, prompt the user according to the user's behavior habits, and prevent the user from forgetting.

[0014] In conjunction with the second aspect, in some implementations, the obtaining of the first text content specifically includes: for the second control, using the getText() method function or the getTooltipText() method function or the getContentDescription() method function to obtain the text content of the second control. Or obtaining the class of the second control; obtaining the declared field attribute according to the class; traversing the declared field attribute, obtaining the field whose declared field attribute is a string attribute, and determining the field as the text content of the second control. Or using the reflection mechanism of JAVA to obtain the return value, and determining the return value as the text content of the second control. Or in the case where the second control is of TextureView type, obtaining the image information of the second control; performing optical character recognition (OCR) processing on the image information to obtain the text content of the second control. Among them, the second control is any one of the controls in the second control set. In this way, the text content within the scope of the first control can be obtained by the above method.

[0015] In combination with the first aspect and the second aspect, in some implementations, after acquiring the first text content and before recommending the first function, the method further includes: displaying a second interface, the second interface including multiple function options and the first text content, the multiple function options including any one or more of the following: a copy option, a paste option, a favorite option, a search option, and a translation option. A second operation for the function option is received, and the operation corresponding to the function option of the second operation is performed on the selected content in the first text content. In this way, the needs of the user can be met, and the functions of copying, pasting, favorites, searching, and translating the text content can be realized.

[0016] In combination with the first aspect and the second aspect, in some implementations, after acquiring the first text content and before recommending the first function, the method further includes: based on the first text content, learning the user's behavior habits, and the user's behavior habits are used to recommend the first function. In this way, the text content can be stored in an electronic device and regular learning can be performed. For example, the electronic device can input the text content into a related learning module for the learning module to perform regular learning, so as to understand the user's behavior habits and better provide the user with relevant recommendation services.

[0017] In combination with the first aspect and the second aspect, in some implementations, the time of recommending the first function is related to the time of receiving the first operation. In some implementations, the above-mentioned method of learning the model and predicting the text content acquired by the electronic device based on time and recommending the content is only an exemplary description and is not limited to a recommendation period of one month or one day. There may be more factors that affect the recommendation period (such as location), and it is necessary to consider the actual behavior habits of users for recommendation.

[0018] In combination with the first aspect and the second aspect, in some implementations, the recommending the first function specifically includes: displaying a third interface, the third interface including a fourth control, the fourth control corresponding to the first function. In some implementations, the electronic device can directly display the first function for the user to select.

[0019] In combination with the first aspect and the second aspect, in some implementations, the third interface is a desktop, and the fourth control is displayed in a recommendation card on the desktop. In some implementations, the recommendation function can be directly displayed on the desktop so that the user can view it in time.

[0020] In combination with the first aspect and the second aspect, in some implementations, the traversal specifically includes: depth-first traversal DFS, so as to ensure that no controls in DecorView are missed.

[0021] In combination with the first aspect and the second aspect, in some implementations, the method further includes: receiving a fifth operation for the first control, and displaying a fourth interface. In some implementations, the electronic device can further obtain text content in the control while implementing a basic function of the control (such as a function of confirming a call control to call a vehicle).

[0022] In a third aspect, the present application provides an electronic device, comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method of the first aspect or any embodiment of the first aspect or the second aspect or any embodiment of the second aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a chip, comprising one or more processors. When the one or more processors execute computer instructions, the electronic device executes the method of the first aspect or any implementation of the first aspect or the second aspect or any implementation of the second aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the above-mentioned first aspect or any implementation method of the first aspect or the second aspect or any implementation method of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A-Figure 2B Various user interfaces displayed by an electronic device provided in an implementation example of the present application.

[0026] Figure 3 A flowchart of a method for obtaining text content provided for an implementation example of the present application.

[0027] Figure 4 A flowchart of another method for obtaining text content provided in an embodiment of the present application.

[0028] Figure 5 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application.

[0029] Figure 6 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0031] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0032] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0033] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0034] When using an electronic device, a user expects to record the text content displayed in the electronic device, share it with others, or search for the text content. However, if the user directly inputs the text content to record, share, or search, the user's operation is too complicated and the user experience is poor. In this case, the electronic device needs to provide a function to obtain text content.

[0035] In some implementations, the electronic device can obtain the text content through the getText method. However, if the control containing the text content is not a TextView type control, the electronic device cannot obtain the text content through the getText method. Moreover, if the control containing the text content is not at the level touched by the user, the electronic device cannot obtain the text content. Furthermore, if the control containing the text content is a button-like type, the user's operation of obtaining the text content may directly trigger a button-click jump event. Generally, the electronic device cannot obtain the text content contained in the button control.

[0036] In order to solve the above problems, the embodiments of the present application provide a method and an electronic device for obtaining text content. In the method, the text content that the user expects to obtain exists in the first area of ​​the interface. The electronic device can obtain the text content through the relevant method functions (such as getTooltipText() method, getContentDescription() method) that can obtain the text content in the first area; the electronic device can also use the reflection mechanism of JAVA to obtain the text content in the first area; if the control containing the text content is not at the level touched by the user, the electronic device can also traverse the parent class controls to determine the controls in the first area that contain text information and are not directly touched by the user, and obtain the text content in the control; if the first area contains a picture, the electronic device can also perform optical character recognition (OCR) text extraction on the picture to obtain the text content in the picture.

[0037] After implementing the method provided by the present application, the problem that the electronic device cannot obtain certain text content is solved. In this way, the electronic device can obtain all the text content displayed in the interface, improve the user experience, and provide protection and convenience for the user to obtain the text content.

[0038] The following introduces relevant concepts involved in the embodiments of the present application.

[0039] JAVA's reflection mechanism: It means that in the running state of the program, you can construct an object of any class, obtain the class to which any object belongs, obtain the member variables and methods of any class, and call the properties and methods of any object. JAVA's reflection mechanism has the function of dynamically obtaining information and dynamically calling objects.

[0040] Optical Character Recognition (OCR): refers to the process by which electronic devices determine the shapes of various characters in an image by detecting dark and light patterns, and then use character recognition methods to convert the shapes into computer text. For example, electronic devices can obtain text content in an image through OCR.

[0041] The following describes the application scenarios of the method for obtaining text content provided by this application.

[0042] In some implementations, the method for obtaining text content provided in the embodiments of the present application can be applied to an electronic device in response to a user selecting text content in a first area in a user interface, and implement functions such as copying, pasting, collecting, searching, and translating the text content.

[0043] In another implementation, the method for obtaining text content provided in the embodiment of the present application can be applied to an electronic device in response to a user selecting a text content in a first area of ​​a user interface, storing the text content in the electronic device, and performing regular learning. For example, the electronic device can input the text content into a related learning module for the learning module to perform regular learning, so as to understand the user's behavior habits and better provide the user with relevant recommendation services.

[0044] The method for obtaining text content provided in the embodiment of the present application can also be applied to more scenarios and is not limited here.

[0045] Next, the user interface schematic diagram under the above scenario provided by the embodiment of the present application is introduced in combination with the UI embodiment.

[0046] Figure 1A The user interface 21 of the electronic device in the scenario of recharging phone charges can be an interface provided by any application with the function of recharging phone charges. The user interface 21 can include but is not limited to a title 211, number information, and phone charge options.

[0047] In some implementations, the electronic device may respond to a user operation on the title 211 by displaying a Figure 1B The user interface 22 shown. The user operation may include but is not limited to: a two-finger press operation, a long press operation, and the like.

[0048] For example, Figure 1A As shown, after the electronic device responds to the user operation on the title 211, the electronic device can obtain the touch point coordinates (e.g., x=200.5, y=181.3), and then obtain the text content of "Mobile Phone Recharge" using the text content acquisition method provided in the embodiment of the present application. For how to obtain the text content, please refer to the relevant description of the following process embodiment, which will not be explained here.

[0049] Figure 1BThe user interface 22 for the electronic device in the scenario of obtaining text content. The user interface 22 may include, but is not limited to: a search control 221, a copy control 222, a translation control 223, text options 224, and a close control 225.

[0050] Among them, the search control 221 can be used to search for the selected text, the copy control 222 is used to copy the selected text to the clipboard for subsequent pasting into other editing areas, the translation control 223 can be used to translate the selected text between different languages, the text options 224 contain multiple selectable texts for responding to user operations to select text. The close control 225 can be used to respond to user operations, exit the user interface 22, and display the user interface 21.

[0051] As Figure 1B shown, the electronic device can respond to the click operations on the text options 224A and 224B by the user, select the two texts "hand" and "machine", and then the electronic device can respond to the click operation on the copy control 222 by the user, and copy the two texts "hand" and "machine" to the clipboard.

[0052] In this way, by obtaining the text content, it is more convenient to meet the user's needs for text, such as copying, pasting, translating, searching, collecting, and other needs.

[0053] In some other implementation manners, the electronic device can also respond to the user operation on the title 211, and store the text content of "mobile phone recharge" in the electronic device.

[0054] Exemplarily, the electronic device can obtain that the user has long-pressed or double-clicked the title 211 on August 26th, and then input the text content of "mobile phone recharge" into the learning model stored in the electronic device for learning. After being trained by the learning model, the electronic device can obtain that the user may need to perform the operation of "mobile phone recharge" on September 26th.

[0055] Exemplarily, on September 26th, the electronic device can display a user interface 23 as Figure 1C shown. The user interface 23 can be the main interface. The user interface 23 includes a recommendation control 231. The recommendation control 231 can be located in the service card. The recommendation control 231 displays the text content of "mobile phone recharge" to prompt the user that they can recharge the phone bill. The electronic device can respond to the click operation on the recommendation control 231 by the user, and jump to display a user interface 21 as Figure 1A shown. The user can perform operations such as inputting the recharge number, selecting the recharge amount, and recharging the phone bill in the user interface 21.

[0056] In this way, the user's behavior habits can be understood by obtaining text content, so as to recommend relevant content to the user and prompt the user according to the user's behavior habits.

[0057] Figure 2A The user interface 31 of the electronic device in the taxi-hailing scenario can be an interface provided by any specific taxi-hailing application. The user interface 31 can include, but is not limited to, a call confirmation control 311, vehicle model options, and a map, etc.

[0058] like Figure 2A As shown, the electronic device can respond to a click operation on the confirm call control 311 to implement a taxi-hailing function.

[0059] In addition to implementing the taxi-hailing function, in a possible implementation, the electronic device can also obtain the text content of "Confirm Call" in the Confirm Call control 311, and implement functions such as copying, pasting, translating, and searching for the text content. For the user interface for implementing the above functions, please refer to Figure 1B Here, how to obtain the text content in the function control can refer to the relevant description of the following process embodiment, which will not be described here.

[0060] In another possible implementation, the electronic device may also store the text content of “Confirm Call” in the confirm call control 311 in the electronic device.

[0061] Exemplarily, the electronic device can obtain that the user clicked the confirm call control 311 at 22:03 on August 26, and then input the text content of "Confirm Call" into the learning model stored in the electronic device for learning. After training the learning model, the electronic device can obtain that the user may need to hail a taxi at about 22:00. Optionally, the electronic device can obtain the text content of "Confirm Call" at multiple times. Exemplarily, the electronic device obtains the text content of "Confirm Call" at around 22:00 every day for a month. At this time, the electronic device can obtain through the learning model that the user may need to hail a taxi at about 22:00.

[0062] For example, at 22:00 on September 26, the electronic device may display the following information: Figure 2B The user interface 32 shown in FIG. 3 may be a main interface. The user interface 32 includes a recommendation control 321. The recommendation control 321 may be located in a service card. The recommendation control 321 may display a text content of "hailing a taxi" to prompt the user to hail a taxi. The electronic device may respond to a user's click operation on the recommendation control 321 and jump to display a Figure 2A As shown in the user interface 31 , the user can select a car model, hail a taxi, etc. in the user interface 31 .

[0063] In this way, the user's behavior habits can be understood by obtaining the text content in the function control, so as to recommend relevant content to the user and prompt the user according to the user's behavior habits.

[0064] In the embodiment of the present application, the above-mentioned text content obtained based on the electronic device, the learning model makes predictions based on time, and the method of recommending content is only illustrative and is not limited to a recommendation period of one month or one day. There may be more factors that affect the recommendation period (such as location), and it is necessary to consider the user's actual behavior habits for recommendation.

[0065] Based on the user interface introduced above, the method flow for obtaining text content provided in an embodiment of the present application is introduced below.

[0066] Figure 3 The flow of the method for acquiring text content provided in the embodiment of the present application is exemplified.

[0067] like Figure 3 As shown, the method flow specifically includes:

[0068] S301: The electronic device detects a first operation acting on a first area. The first interface includes a first area, and the first area includes text information.

[0069] In some implementations, the first area may refer to a certain range of text content that the user expects to obtain in the first interface, or may refer to a certain range of the first interface where the first operation is detected. In other words, the first operation may act on any area containing text content in the first interface.

[0070] In a possible implementation, the first operation may also include an operation of switching from other interfaces to the first interface, and the scope of the first area may refer to the entire first interface.

[0071] The first operation may include, but is not limited to: a two-finger press operation, a long press operation, a slide operation, a voice command, a gesture command, etc. The first interface may include a user interface including text content provided by any application.

[0072] Optionally, the first area may be slightly larger than the range covered by the text content. In this application, slightly larger is a relative concept rather than a specific numerical limit. This allows the user to obtain the text content even if the user does not completely touch the range covered by the text content, avoiding excessive restrictions on the position touched by the user in the interface, thereby improving the user experience.

[0073] In some implementations, after the electronic device detects the first operation acting on the first area, the coordinate position of the first operation can be obtained. The coordinate position can be used to determine the first control set in the subsequent step S303, which is not described here. In the embodiment of the present application, the coordinate position of the first operation can be referred to as the first coordinate.

[0074] Since the position touched by the user may not only be a point but a range, in a possible implementation, the electronic device may determine the center position of the range touched by the user when performing the first operation as the user touch position, and the coordinates of the center position are the user touch coordinates.

[0075] For example, after the electronic device detects the first operation of the user on the title 211, the electronic device can obtain the user touch coordinates, which can be expressed as X=200.5, Y=181.3. The user touch coordinates are obtained based on the upper left corner of the user interface 21 as the coordinate origin.

[0076] In the embodiment of the present application, the above user touch coordinates are only for exemplary description, and the user touch coordinates are only used to indicate the specific position of the first operation in the first interface. The user touch coordinates can be obtained based on any coordinate origin, and there is no limitation on this.

[0077] For example, the first region may include Figure 1A The scope covered by Title 211, namely Figure 1A The first interface may include a user interface 21. The first operation may include a two-finger press operation, a long press operation, etc. for the title 211. In the embodiment of the present application, the first area, the first interface, and the first operation are only illustrative and not limited thereto.

[0078] In a possible implementation, after detecting the first operation, the electronic device may also detect the user's operation of selecting the area to be obtained for text content, and determine the area to be obtained for text content according to the range selected by the user. This is because after the user performs the first operation, the text content in the position touched by the first operation may not be what the user expects, and the electronic device may also determine the area to be obtained for text content according to the user's needs.

[0079] Exemplarily, after responding to the first operation, the electronic device may display a selection box, and the text content in the interface range covered by the selection box may not be what the user expects. Then the electronic device may change the position and size of the selection box in response to the user's sliding operation or dragging operation on the selection box. The range covered by the changed selection box is the area where the text content is to be obtained. The text content in the area where the text content is to be obtained may be what the user expects. In this way, the user can also change the area where the text content is to be obtained according to their own needs, which is more flexible.

[0080] In the embodiment of the present application, the multiple controls in the above selection box can be referred to as a third control set.

[0081] S302: The electronic device obtains a root node view (DecorView) of the first interface.

[0082] In some implementations, after the electronic device responds to the first operation, it obtains the DecorView of the first interface. Among them, DecorView, as a subclass of frame layout (FrameLayout), can be understood as the root node view of the Android view tree. DecorView is the top-level view (view) in the window (window). The electronic device can obtain DecorView from the window of the first interface. For example, the electronic device can obtain DecorView through the method of getWindow().getDecorView(). In the embodiments of the present application, there may be more ways to obtain the DecorView of the first interface, which is not limited to this.

[0083] DecorView has only one child View, which is a vertical linear layout (LinearLayout). It contains two child elements, one of which is the title view (TitleView), which is the container of the action bar (ActionBar), and the other is the content view (ContentView), which is the container of the content in the window.

[0084] Optionally, the electronic device can also obtain the area displayed by the first interface through the getWindowVisibleDisplayFrame method, which may include the title bar and the content bar, but does not include the status bar. In general, the first area affected by the first operation does not include the range covered by the status bar, but may include the range covered by the title bar and the content bar.

[0085] S303: The electronic device traverses the DecorView of the first interface and determines a first control set, wherein the first control set is a control set in the first area.

[0086] In some implementations, the purpose of the electronic device acquiring the DecorView of the first interface is to determine the control to which the text content belongs. Therefore, after the electronic device acquires the DecorView of the first interface, it traverses all the controls in the DecorView to determine the control set located in the first area. In an embodiment of the present application, the control set in the first area is referred to as the first control set.

[0087] Specifically, the electronic device can use a depth-first search (DFS) method to traverse the DecorView in the first interface. DFS is actually a recursive process, which can also be understood as a pre-order traversal of a tree, which can ensure that no controls in the DecorView are missed. In the embodiment of the present application, other traversal methods that can satisfy the determination of the control collection in the first area can also be used to traverse the DecorView, which is not limited.

[0088] In some implementations, the electronic device determines the first control set by traversing DecorView, which actually obtains the relevant information of all sub-controls in DecorView, and then determines which controls are located in the first area based on the relevant information. Exemplarily, the electronic device can obtain the coordinate information of all sub-controls in the first interface, and then determine whether the coordinates of each sub-control are located in the first area, and finally determine one or more controls located in the first area.

[0089] For example, the electronic device can obtain the left, top, right, and bottom attributes of the child control. Among them, top can represent the distance from the top edge of the child control to the top edge of its parent layout; left represents the distance from the left side of the child control to the left side of its parent layout; right represents the distance from the right side of the child control to the left side of its parent layout; bottom represents the distance from the bottom edge of the child control to the bottom edge of its parent layout. In some implementations, the above-mentioned parent layout may refer to a window, and the top, left, right, and bottom edges of the parent layout may refer to the top, left, right, and bottom edges of the window. In an embodiment of the present application, the left, top, right, and bottom attributes can be used to represent the position information of the child control.

[0090] Then, after obtaining the left, top, right, and bottom attributes, the electronic device can obtain the coordinates of the subcontrol. Since the subcontrol has a certain size, the coordinates of the subcontrol can include a coordinate range. Exemplarily, the coordinates of the subcontrol A can be expressed as X=195.3~201.5, Y=180.1~183.6. The coordinates of the subcontrol A can be obtained based on the upper left corner of the user interface to which the subcontrol A belongs as the coordinate origin.

[0091] In the embodiment of the present application, the coordinates of the above sub-control A are only used for exemplary description, and the coordinates of the sub-control A are only used for the specific position of the sub-control A in the interface. The coordinates of the sub-control A can be obtained based on any coordinate origin, and there is no limitation on this. In the embodiment of the present application, the electronic device can also obtain the coordinates of the sub-control in other ways, not limited to the above method.

[0092] In some implementations, after acquiring the coordinates of all sub-controls, the electronic device can filter out one or more sub-controls from all sub-controls in combination with the user touch coordinates acquired in S301, and the coordinates of the one or more sub-controls include the user touch coordinates. When the electronic device filters out multiple sub-controls, the multiple sub-controls can be referred to as a control set in the first area.

[0093] S304: The electronic device traverses the first control set and obtains text content in the first area.

[0094] In some implementations, the above-mentioned traversal of the first control set may specifically refer to the electronic device using the DFS method to traverse the control set in the first area. DFS is actually a recursive process, which can also be understood as a pre-order traversal of the tree, which can ensure that the controls in the first control set are not missed. In the embodiment of the present application, the traversal method can also be used to traverse the first control set, which is not limited to this.

[0095] In some implementations, the electronic device traverses the first control set and obtains the text content in the first area, which actually means that the electronic device performs a text content acquisition operation for each control obtained by traversing the first control set, until all controls in the first control set have performed the text content acquisition operation. The first control set may contain controls that do not contain text content, and may contain controls that contain text content. It is worth noting that performing a text content acquisition operation may not necessarily obtain the text content. If a control itself does not contain text content, then performing the text content acquisition operation will not obtain the text content.

[0096] The following specifically introduces several methods for obtaining the text content in the first area, that is, methods for performing the operation of obtaining the text content:

[0097] Method 1: The electronic device obtains text content by using a method function for the control, which may include but is not limited to: getTooltipText() method, getContentDescription() method, etc.

[0098] Tooltip may refer to a prompt box component. For example, when the type of control B in the first control set is a prompt box component, the electronic device may directly obtain the text content in control B through the getTooltipText() method. ContentDescription may refer to adding an explanation of the control for people with visual impairments, for example, the text content may be played out by voice.

[0099] If the electronic device cannot obtain the text content through the above method function, it means that the control type is not the above two types, and then it is determined whether the type of the control is a text control (TextView).

[0100] In the embodiment of the present application, the types of controls may include but are not limited to: basic controls, advanced controls and custom controls. Among them, basic controls may include: text controls (TextView), button controls (Button), image display controls (ImageView), and text editing controls (EditView); advanced controls may include: list controls (ListView), recycling controls (RecycleView), and web page controls (WebView).

[0101] When the electronic device determines that the control type is TextView, the text content can be obtained through the getText() method. When the electronic device determines that the control type is not TextView, the following method can be used to obtain the text content. In other words, if the electronic device cannot obtain the text content through method one, the text content can be obtained through method two.

[0102] Method 2: The electronic device obtains the text content by traversing the declared fields (declaredFields) attribute of the control and then obtaining the corresponding field (field).

[0103] Specifically, the first step: the electronic device obtains the class (class) of the control, and then obtains the declaredFields attribute collection corresponding to the control through the class. The second step: the electronic device traverses the declaredFields attribute collection, and obtains the value of the field whose class is the string class (java.lang.String). Here, the value of the field is not empty. The third step: the electronic device traverses the class to the upper layer, obtains the parent class of the class, and loops the first and second steps above until the class is empty or the class no longer corresponds to the control type. In some implementations, the electronic device can use the getClass() method to obtain the class of the control. The electronic device can obtain the value of the field through the getFields() method and the getDeclaredFields() method.

[0104] The getFields() method can be used to obtain all public fields of a class, and can also obtain fields in the parent class.

[0105] The getDeclaredFields() method can be used to obtain all declared fields of a class, that is, public, private, and protected fields, but does not include fields in the parent class.

[0106] In the embodiment of the present application, in order to avoid the situation where the text content in the first area is defined as private, the getDeclaredFields() method is used to obtain the field.

[0107] In the embodiment of the present application, the use of method 2 requires the electronic device to perform traversal, and the time and resources required are more than those of method 1. Therefore, method 2 is more complicated than method 1.

[0108] When the electronic device cannot obtain any text content through method two, that is, when there is no field whose class is java.lang.String, the electronic device can obtain the text content through method three.

[0109] Method 3: The electronic device can obtain the return value by calling the reflection mechanism of JAVA, thereby obtaining the text content.

[0110] Specifically, when the electronic device obtains the control name and method name of the control, the reflection method can be used to directly obtain the text content. The premise of method three is that some or all elements in the first interface are open to the electronic device. For example, the first interface can be an interface in a system application. If the first interface is an interface of a third-party application, and the electronic device can obtain the specific name and specific method of the control, method three can also be used to obtain the text content.

[0111] In the embodiment of the present application, the method three is actually a method for executing the control. That is, the electronic device directly executes the program corresponding to the method. This will consume the resources of the electronic device, prolong the time to obtain the text content, and may also cause the page to jump. Therefore, the method three is more complicated than the method two.

[0112] When the electronic device cannot obtain any text content through method three, the text content can be obtained through method four.

[0113] Method 4: When the electronic device determines that the control is of the TextureView type, the method function for the control is used to obtain image information. For example, the getBitmap() method can be used to obtain image information. Then the image information is processed by OCR to obtain the text content. This can better avoid privacy leakage than directly taking a screenshot of the image type control and then obtaining the text content.

[0114] When the electronic device determines that the control is another type of image control, the electronic device may not be able to obtain the text content through the getBitmap() method. In this case, the text content can also be obtained by taking a screenshot. In this way, the electronic device can obtain the text content for various types of controls containing text content.

[0115] Since the resources consumed by the electronic device implementing methods one to four in sequence gradually increase, the electronic device adopts methods one to four to obtain the text with priority. Specifically, the priority of methods one to four gradually decreases. The electronic device can first use a high priority method to obtain the text content for one or more controls. After the text content fails to be obtained, the electronic device uses the second highest priority method, and so on, until the text content is successfully obtained.

[0116] Exemplarily, the first control set may include control a, control b, and control c.

[0117] In a possible implementation, the electronic device may sequentially use method 1, method 2, method 3, and method 4 to obtain text content for control a, but all of them fail. Then the electronic device sequentially uses method 1, method 2, method 3, and method 4 to obtain text content for control b. After the electronic device uses method 2 to obtain text content for control b, the traversal stops, and the text content obtained by using method 2 for control b is the text content in the first area.

[0118] In another possible implementation, the electronic device may use method 1 to sequentially obtain text content from control a, control b, and control c, but all of them fail to obtain. Then the electronic device uses method 2 to sequentially obtain text content from control a, control b, and control c. When the electronic device successfully obtains text content from control b using method 2, the traversal is stopped, and the text content obtained from control b using method 2 is the text content in the first area.

[0119] In the embodiment of the present application, the above sequence of methods for obtaining text content for each control in the first control set is only for illustrative purposes and is not intended to be limiting.

[0120] In a possible implementation, after executing S303, when the number of controls acquired by the electronic device in the first area is only one, the electronic device may also sequentially adopt methods 1 to 4 to acquire text content for only one control in the first area.

[0121] In S304, once the electronic device obtains any text content through any of the above four methods, the step can be stopped, and the obtained text content is the text content in the first area.

[0122] After executing S304, the electronic device may execute the following S305-1 alone, or the following S305-2 alone, or execute S305-1 and S305-2 simultaneously. In the embodiment of the present application, it depends on the actual needs of the user and is not limited to this.

[0123] S305-1. The electronic device may store the acquired text content in the electronic device.

[0124] After the electronic device acquires the text content, the text content can be input into the learning model stored in the electronic device for regular learning. After learning the text content, the electronic device can make corresponding content recommendations at a specific time, specific location, etc.

[0125] For example, after the electronic device obtains the text content "mobile phone recharge" at time a, it stores the text content "mobile phone recharge" in the learning model for regular learning. Then the electronic device can recommend the user to perform related operations for mobile phone recharge at time b, where time a and time b can be about one month apart. Figure 1C In some implementations, the user interface for recommending the first function may be referred to as the third interface, and the control for recommending the first function may be referred to as the fourth control. For example, the third interface may specifically refer to the user interface 23.

[0126] In this way, the electronic device can understand the user's behavior habits by acquiring text content, so as to recommend relevant content to the user, predict the user's operation according to the user's behavior habits and prompt the user to prevent the user from forgetting.

[0127] S305-2. The electronic device can copy, paste, save, search, translate, and perform other functions on the acquired text content.

[0128] After the electronic device obtains the text content, it can respond to the user's second operation on the text content and implement various functions on the text content, which may include but are not limited to: copying, pasting, collecting, searching, translating, etc. of the obtained text. Figure 1B In the embodiment of the present application, the user interface that can be used to implement the above-mentioned various functions for text content can be referred to as the second interface. For example, the second interface can refer to the user interface 22.

[0129] Exemplarily, the second operation may include a click operation on the search control 221, the copy control 222, and the translation control 223. In the embodiment of the present application, the second operation is only exemplified and is not limited thereto.

[0130] In this way, the electronic device can obtain text content and more conveniently meet the user's needs for text content, such as copying, pasting, translating, searching, collecting, etc. Especially when there is a lot of text content, the user does not need to input it by himself, which can better reflect the convenience.

[0131] Figure 4 The flow of another method for acquiring text content provided in an embodiment of the present application is exemplified.

[0132] like Figure 4 As shown, the method flow specifically includes:

[0133] S401: The electronic device detects a third operation acting on a first control.

[0134] In some implementations, the first control is different from each control in the control set in the first area, and the first control is a control at the top level. The electronic device can jump to other user interfaces in response to the user's operation on the first control. That is, the first control can be used to implement a specific function. In an embodiment of the present application, the fifth operation may refer to a click operation, and the fourth interface may refer to other user interfaces to which the jump occurs, such as a user interface of a successful clock-in displayed by the electronic device after the user clicks the clock-in button. The first control may be a functional control or a functional button, such as a clock-in button or a taxi button. The third operation may include, but is not limited to, a click operation, a double-click operation, a slide operation, a long press operation, a voice command, a gesture command, and the like.

[0135] Exemplarily, the first control may include Figure 2A The third operation may include a click operation for the confirmation call control 311. In the embodiment of the present application, the first control and the third operation are only exemplified and are not limited thereto.

[0136] S402: The electronic device obtains text content in the first control.

[0137] In some implementations, the electronic device may obtain text content in the first control after responding to a third operation performed by the user on the first control.

[0138] For the specific method of obtaining the text content in the first control, reference may be made to the related descriptions of method 1 to method 4 for obtaining the text content in the above S304, which will not be elaborated here.

[0139] In some implementations, if the electronic device acquires text content using the above method 1, method 2, method 3, or method 4, it indicates that the first control contains text content. At this time, the electronic device may execute the following S405-1 and / or S405-2.

[0140] In some implementations, if the electronic device fails to obtain text content after using the above methods 1, 2, 3, and 4, it indicates that the first control does not contain text content, that is, the text content in the first control is empty. The text observed by the user within the range of the first control may be contained in other child controls of the parent control on the upper layer of the first control. At this time, the electronic device can execute the following S403.

[0141] S403: When the text content in the acquired first control is empty, the electronic device acquires the parent control of the first control.

[0142] In some implementations, when the first control does not contain text content, the electronic device determines that the text content may exist in other child controls of the parent control of the first control. Therefore, the electronic device may first obtain the parent control of the first control.

[0143] Specifically, the electronic device can use the getParentObject() method to traverse and find whether the parent control of the first control exists, and then use the getParent() method to obtain the parent control of the first control. The getParent() method can obtain the parent control according to the path. Generally, the number of parent controls of the first control is one.

[0144] S404: The electronic device obtains a second control set, and traverses the second control set to obtain text content, wherein the second control set is a child control set of the parent control of the first control.

[0145] In some implementations, after the electronic device obtains the parent control of the first control, it can obtain the child control based on the parent control.

[0146] Specifically, the electronic device can traverse and find whether the child control of the parent control exists through the getChildObject() method, and then obtain the child control set of the parent control through the getChildObjects() method. The number of child controls of the parent control can be one or more. When the electronic device obtains multiple child controls of the parent control, the multiple child controls of the parent control of the first control can be referred to as the second control set.

[0147] In some implementations, the above-mentioned traversal of the second control set may specifically refer to the electronic device using the DFS method to traverse the child control set of the parent control. DFS is actually a recursive process, which can also be understood as a pre-order traversal of the tree, which can ensure that no controls in the second control set are missed. In the embodiment of the present application, the traversal method can also be used to traverse the second control set, which is not limited to this.

[0148] In some implementations, the electronic device traverses the second control set and obtains text content actually means that the electronic device traverses the second control set and obtains the controls (i.e., third controls) that are within the range covered by the first control, and performs a text content acquisition operation on one or more third controls until all third controls in the second control set have performed the text content acquisition operation.

[0149] The second control set may contain a third control that does not contain text content, or may contain a third control that contains text content. It is worth noting that executing the text content acquisition operation may not necessarily obtain the text content. If a third control itself does not contain text content, executing the text content acquisition operation will not obtain the text content.

[0150] For the specific method of obtaining text content, please refer to the description of the method 1 to the method 4 for obtaining text content in the above S304, which will not be repeated here. For the specific method of the order of obtaining the text content in the third control in the second control set, please refer to the description of the order of the method 1 to the method 4 used for obtaining text content in the above S304, which will not be repeated here.

[0151] In a possible implementation, after executing S403, when the number of third controls acquired by the electronic device is only one, the electronic device may also sequentially adopt method one to method four to acquire text content for only one third control.

[0152] After executing S404, the electronic device may execute S405-1 alone, or S405-2 alone, or execute S405-1 and S405-2 simultaneously. In the embodiment of the present application, it depends on the actual needs of the user and is not limited to this.

[0153] S405-1. The electronic device may store the acquired text content in the electronic device.

[0154] After the electronic device acquires the text content, the text content can be input into the learning model stored in the electronic device for regular learning. After learning the text content, the electronic device can make corresponding content recommendations at a specific time, specific location, etc.

[0155] For example, after the electronic device obtains the text content "Confirm call" at time c, it stores the text content "Confirm call" into the learning model for regular learning. Then the electronic device can recommend the user to take a taxi-related operation at time d, where time c and time d can be about one day apart. Figure 2B The relevant description is not repeated here.

[0156] In this way, the electronic device can understand the user's behavior habits by obtaining the text content contained in the function control, so as to recommend relevant content to the user, predict the user's operation according to the user's behavior habits and prompt the user to prevent the user from forgetting.

[0157] S405-2. The electronic device can copy, paste, save, search, translate, and perform other functions on the acquired text content.

[0158] After the electronic device obtains the text content, it can respond to the user's second operation on the text content and implement various functions on the text content, which may include but are not limited to: copying, pasting, collecting, searching, translating, etc. of the obtained text. Figure 2A The relevant description is not repeated here.

[0159] Exemplarily, the second operation may include a click operation on the search control 221, the copy control 222, and the translation control 223. In the embodiment of the present application, the second operation is only exemplified and is not limited thereto.

[0160] In one possible implementation, the electronic device may jump to an interface for implementing the function of the first control after responding to the third operation on the first control, and display various function options for the acquired text content in the interface after the jump.

[0161] In another possible implementation, the electronic device may directly display various function options for the acquired text content after responding to the third operation on the first control, and jump to the interface for implementing the function of the first control after responding to the user's operation on the function option or canceling the operation of selecting the function option.

[0162] In the embodiment of the present application, the order of displaying various function options for text content is not limited.

[0163] In this way, the electronic device can obtain text content and more conveniently meet the user's needs for text content, such as copying, pasting, translating, searching, collecting, etc. Especially when there is a lot of text content, the user does not need to input it by himself, which can better reflect the convenience.

[0164] In a possible implementation, when the electronic device detects that the first area does not contain any functional controls, that is, the first area only contains text content for display, the electronic device may execute Figure 3 When the electronic device detects that the first area contains a function control, that is, the first area only contains a function control for jumping to other interfaces, the electronic device can execute Figure 4 In another possible implementation, when the first area contains both text content for display only and function controls, the electronic device can simultaneously use Figure 3 as well as Figure 4 The two methods of text content shown are not limited in the embodiments of the present application.

[0165] In the embodiment of the present application, the text content finally obtained by the above-mentioned flowcharts can be referred to as the first text content.

[0166] In the embodiment of the present application, the above-mentioned content recommendation based on the text content can be referred to as recommending the first function, and the first function is, for example, a mobile phone recharge function, a taxi-hailing function, and the like.

[0167] refer to Figure 5 , Figure 5 A schematic structural diagram of an electronic device 100 is shown.

[0168] The electronic device 100 may be equipped with Or a portable terminal device of other operating systems, the electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle, a vehicle-mounted device, a smart home device and / or a smart city device, etc. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0169] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0170] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0171] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0172] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0173] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0174] In some implementations, the memory may be used to store the acquired text content.

[0175] In some implementations, the processor 110 may be used to obtain the coordinate position of the first operation. The processor 110 may also be used to obtain the DecorView of the first interface. The processor 110 may also be used to obtain the control set in the first area. The processor 110 may also be used to obtain the text content of the first area according to the above methods 1 to 4. The processor 110 may also be used to obtain the text content in the first control.

[0176] In some implementations, the processor 110 may be used to obtain text content in the first control. The processor 110 may be used to obtain the parent control of the first control. The processor 110 may be used to obtain a set of child controls of the parent control of the first control. The processor 110 may be used to obtain text content by traversing the second control set.

[0177] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0178] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0179] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0180] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0181] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0182] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0183] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0184] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0185] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0186] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0187] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0188] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0189] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0190] In the embodiment of the present application, the electronic device can display, for example, the following information provided in the above UI embodiment through the display screen 194: Figure 1A-Figure 2B The user interface shown and so on will not be described in detail here.

[0191] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0192] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0193] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0194] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0195] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0196] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0197] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.; non-volatile memory may include disk storage devices and flash memory.

[0198] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0199] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0200] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0201] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0202] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0203] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0204] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0205] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0206] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0207] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0208] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0209] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0210] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0211] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0212] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0213] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0214] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0215] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0216] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0217] In some implementations, the touch sensor 180K can be used to detect a user's operation on the first area or the first control, and can also detect a second operation of selecting a text option.

[0218] The bone conduction sensor 180M can obtain a vibration signal. The buttons 190 include a power button, a volume button, etc. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect a SIM card.

[0219] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0220] Figure 6 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0221] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0222] The application layer can include a series of application packages. Figure 6 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0223] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 6 As shown, the application framework layer may include an input manager (InputManager), a window manager, a content provider, a view system (ViewSystem), a phone manager, a resource manager, a notification manager, and the like.

[0224] The InputManager can be used to monitor the user's input operations, for example, the user's first operation, second operation, third operation, etc. By monitoring the user's input operations, the electronic device 100 can determine the specific location of the user's operation in the user interface and the specific control affected.

[0225] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0226] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0227] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0228] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0229] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0230] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0231] In some implementations, the application framework layer may further include: a view service (ViewService), a view system (ViewSystem) and a view server (ViewServer). Among them, ViewServer includes a view manager (SurfaceFlinger) and an algorithm module.

[0232] Among them, ViewService is a system-level view service, which is used to provide display interface related services for various applications. ViewService can be used to detect various business requests for obtaining text content, and respond to the request to start scheduling related modules to perform corresponding tasks. For example, after InputManager receives the user's first operation, ViewService determines that the first operation is used to obtain the text content of the first area, and notifies ViewSystem of the request to obtain text content. For another example, after InputManager receives the user's third operation, ViewService determines that the third operation is used to obtain the text content of the first control, and notifies ViewSystem of the request to obtain text content.

[0233] Among them, ViewSystem is equivalent to the inner class of ViewService, which can be called by ViewService as a bridge for interaction between the Java layer and the native layer. Specifically, it can be used to receive the information sent by ViewService to obtain the text content.

[0234] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0235] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0236] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0237] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0238] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0239] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0240] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0241] A 2D graphics engine is a drawing engine for 2D drawings.

[0242] The media library can include functional modules such as ViewSystem and ViewService corresponding to the Framework layer.

[0243] Most of the classes in the Framework layer are just "intermediaries" for applications to use library files in the system library. Because the upper-level applications are written in Java, they need the most direct support of Java interfaces, and the system library supports the operation of another language (such as C++). Therefore, the Framework layer acts as an intermediary between the application layer and the system library. Each module in the Framework layer does not really implement specific functions, or only implements part of the functions, and focuses on the core library to complete. For example, ViewSystem and ViewService in the Framework layer also have corresponding classes in the system library. The difference is that the Framework layer is mostly written in Java, and most of the system library is written in C++.

[0244] In some implementations, after ViewSystem obtains the request for obtaining the text content of the first area, it calls the window manager to obtain the DecorView in the first interface where the first area is located. Then, ViewService obtains the first control set by traversing the DecorView. ViewService then calls the function functions contained in the core library (such as getTooltipText(), getContentDescription(), getText(), getDeclaredFields(), getBitmap()) based on methods one to four in S304 above to obtain the text content of the first area. Then, ViewService can return the text content of the first area to the view system through the display driver, so that the electronic device displays as shown. Figure 1B The interface shown; the text content of the first area can also be stored in the database of the system library for the learning module stored in the database to call for regular learning.

[0245] In other implementations, after ViewSystem obtains the request for obtaining the text content of the first control, ViewService directly calls the function functions (such as getTooltipText(), getContentDescription(), getText(), getDeclaredFields(), getBitmap()) contained in the core library based on the above-mentioned methods 1 to 4 in S304 to obtain the text content of the first control. After the acquisition fails, ViewService can call the function function (such as getParent(), getChildObjects()) to obtain the child control set of the parent control of the first control. Then ViewService calls the function functions (such as getTooltipText(), getContentDescription(), getText(), getDeclaredFields(), getBitmap()) contained in the core library based on the above-mentioned methods 1 to 4 in S304 to obtain the text content. Then ViewService can return the obtained text content to the view system through the display driver, so that the electronic device displays an interface containing the text content of "Confirm Call"; it can also store the obtained text content in the database of the system library for the learning module stored in the database to call for regular learning.

[0246] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0247] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 (including the transmission of data flow and control flow) in conjunction with the capture and photo shooting scene.

[0248] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0249] Understandably, Figure 6 The illustrated structure does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0250] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0251] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0252] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0253] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0254] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0255] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0256] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0257] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0258] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0259] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0260] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0261] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0262] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0263] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for obtaining text content, characterized in that: The method comprises: Receiving a first operation on a first area in a first interface; Obtaining the root node view of the first interface; Traversing the root node view of the first interface to determine a first control set, where the first control set includes controls in the first area; Acquire first text content, where the first text content includes text content of controls in the first control set; A first function is recommended, where the first function is related to the first text content.

2. The method according to claim 1, characterized in that After obtaining the first text content, the method further includes: Display a selection box, the range of which includes the first area; receiving a fourth operation for the selection box, wherein the fourth operation is used to change a position and / or a size of the selection box; Traversing the root node view of the first interface to determine a third control set, where the third control set includes controls in the selection box; A third text content is obtained, where the third text content includes text content of controls in the third control set.

3. The method according to claim 1 or 2, characterized in that: After receiving the first operation on the first area in the first interface, the method further includes: Acquire a first coordinate of the first operation in the first interface; Determining the first control set specifically includes: Obtaining the coordinate ranges of multiple controls in the root node view; A control in the root node view whose coordinate range includes the first coordinate is determined as a control in the first control set.

4. The method according to claim 3, characterized in that The obtaining of the coordinate ranges of the multiple controls in the root node view specifically includes: Obtaining location information of multiple controls in the root node view; The coordinate range is acquired according to the position information of the multiple controls.

5. The method according to any one of claims 1 to 4, characterized in that The obtaining of the first text content specifically includes: For the second control, use the getText() method function or the getTooltipText() method function or the getContentDescription() method function to obtain the text content of the second control; or Obtain the class of the second control; obtain the declared field attribute according to the class; traverse the declared field attributes, obtain the field whose declared field attribute is a string attribute, and determine the field as the text content of the second control; or Using the reflection mechanism of JAVA to obtain the return value, and determining the return value as the text content of the second control; or In the case where the second control is of TextureView type, image information of the second control is obtained; optical character recognition (OCR) processing is performed on the image information to obtain text content of the second control; The second control is any one of the controls in the first control set.

6. A method for obtaining text content, characterized in that: The method comprises: receiving a first operation on a first control; Get the parent control of the first control; Acquire a second control set, where the second control set includes child controls of the parent control; Acquire a first text content, where the first text content includes text contents of controls in the second control set that are within a range covered by the first control; A first function is recommended according to the text content, where the first function is related to the first text content.

7. The method according to claim 6, characterized in that The obtaining of the first text content specifically includes: For the second control, use the getText() method function or the getTooltipText() method function or the getContentDescription() method function to obtain the text content of the second control; or Obtain the class of the second control; obtain the declared field attribute according to the class; traverse the declared field attributes, obtain the field whose declared field attribute is a string attribute, and determine the field as the text content of the second control; or Using the reflection mechanism of JAVA to obtain the return value, and determining the return value as the text content of the second control; or In the case where the second control is of TextureView type, image information of the second control is obtained; optical character recognition (OCR) processing is performed on the image information to obtain text content of the second control; The second control is any one control in the second control set.

8. The method according to any one of claims 1 to 7, characterized in that After acquiring the first text content and before recommending the first function, the method further includes: Displaying a second interface, the second interface including a plurality of function options and the first text content, the plurality of function options including any one or more of the following: a copy option, a paste option, a favorite option, a search option, and a translation option; A second operation for the function option is received, and an operation corresponding to the function option for which the second operation is received is performed on the content selected in the first text content.

9. The method according to any one of claims 1 to 8, characterized in that After acquiring the first text content and before recommending the first function, the method further includes: Based on the first text content, the user's behavior habits are learned, and the user's behavior habits are used to recommend the first function.

10. The method according to claim 9, characterized in that The time of recommending the first function is related to the time of receiving the first operation.

11. The method according to any one of claims 1 to 10, characterized in that The recommended first function specifically includes: A third interface is displayed, wherein the third interface includes a fourth control, and the fourth control corresponds to the first function.

12. The method according to claim 11, characterized in that The third interface is a desktop, and the fourth control is displayed in a recommendation card on the desktop.

13. The method according to any one of claims 1 to 12, characterized in that The traversal specifically includes: depth-first traversal DFS.

14. The method according to claim 7 or 8, characterized in that The method further comprises: A fifth operation for the first control is received, and a fourth interface is displayed.

15. An electronic device, characterized in that: The electronic device includes a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in any one of claims 1-14.

16. A chip, applied to electronic equipment, characterized in that: The chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-14.

17. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 14.