Page data processing method and electronic equipment

Abstract

The invention provides a page data processing method and electronic equipment, and the method comprises the steps: determining the node characteristics of a node when the downloading completion information or version updating completion information of an application is received; determining candidate nodes from the nodes of the to-be-processed page based on the first type of node features; determining function nodes from the candidate nodes based on the second type of node features; determining a function node file corresponding to the to-be-processed page based on a function node in the to-be-processed page; the function node file comprises a part of or all data fields of the function node; based on the function node file and a preset layout parameter, adjusting a display size and a display position of the function node in a preset display area, and determining a page layout corresponding to the to-be-processed page; wherein the first type of node features comprise node widths and node heights of nodes; the second type of node features comprise structural features and node attributes of the nodes. According to the invention, the function node identification efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a page data processing method and electronic device. Background Art

[0002] With the continuous development of electronic technology, there are more and more types of electronic devices, and the differences in the display screens of electronic devices are becoming more and more obvious. For example, an electronic device can be a straight-screen mobile phone, a folding-screen mobile phone, or a tablet computer, etc. The display parameters of the display screens of different electronic devices, such as size or resolution, may be different. Under different display parameters, even for the same page, the content displayed on the screen may be different. For example, the screen size of a folding-screen mobile phone in the folded state is different from the screen size in the unfolded state. Therefore, in the folded state, the number of application icons that can be displayed on the screen is different from the number of application icons that can be displayed on the screen in the unfolded state.

[0003] In order to ensure that the electronic device can fully display the page content of the page, or to ensure the integrity and consistency of the page content displayed on the screen when the display parameters of the display screen of the electronic device change, it is usually necessary to manually identify the characteristic information of the page of each application, for example, the characteristic information of each display area corresponding to the page on the screen, such as the size and position of the display area, and the number, size, position and other characteristic information of the display elements (such as icons, text, etc.) included in the display area, so that the developer can formulate a page layout plan based on the characteristic information of the page and apply the page layout plan to the electronic device. The electronic device can control the display screen to display based on the page layout plan. For each different page, the characteristic information of the page needs to be manually identified separately, which increases the workload of the developer.

[0004] Therefore, how to quickly and accurately identify the feature information of a page is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a page data processing method and an electronic device, which can improve the efficiency and accuracy of identifying functional nodes in a page.

[0006] In a first aspect, an embodiment of the present application provides a page data processing method. This method can be executed by an electronic device, or by a device matching the electronic device, such as being executed by a processor, a chip, or a chip system, etc. The method may include: when receiving the download completion information or version update completion information of an application, based on the page data of the page to be processed of the application, determining the node features of the nodes in the page to be processed; the node features include a first type of node features and a second type of node features; the first type of node features includes the node width and node height of the node; the second type of node features includes the structural features and node attributes of the node; based on the first type of node features, determining candidate nodes from the nodes of the page to be processed; based on the second type of node features, determining functional nodes from the candidate nodes; based on the functional nodes, determining a functional node file corresponding to the page to be processed; the functional node file includes some or all of the data fields of the functional nodes; based on the functional node file and preset layout parameters, adjusting the display size and display position of the functional nodes in a preset display area to determine the page layout corresponding to the page to be processed; the preset display area is the area on the screen for displaying the page to be processed; the preset layout parameters include the width and height of the preset display area.

[0007] It can be seen that based on the node features in the page to be processed, the present application can automatically determine the functional nodes among the nodes of the page to be processed, improving the accuracy and efficiency of functional node recognition. Outputting the data fields of the nodes determined as functional nodes is beneficial for subsequent adjusting the page layout of the page to be processed using the functional node file.

[0008] In a possible implementation manner, determining candidate nodes from the nodes of the page to be processed based on the first type of node features includes: determining, from the nodes of the page to be processed, candidate nodes whose node height is greater than a first threshold and whose node width satisfies a first condition; where candidate node i is any one of the candidate nodes, and the node width of candidate node i satisfying the first condition includes one or more of the following: the node width of candidate node i is equal to the screen width; the node width of candidate node i is equal to one-half of the screen width; the node width of candidate node i is greater than or equal to three-fourths of the screen width.

[0009] It can be seen that based on the node height and node width of the nodes, candidate nodes satisfying the first condition can be determined from the nodes of the page to be processed, which is beneficial for accurately identifying functional nodes.

[0010] In a possible implementation manner, determining functional nodes from the candidate nodes based on the second type of node features includes: if the structural features of candidate node p indicate that candidate node p has no child nodes and the node attribute of candidate node p is a text attribute, determining candidate node p as a functional node; candidate node p is any one of the candidate nodes in the page to be processed.

[0011] It can be seen that when a candidate node has no child nodes, it is possible to determine whether the candidate node is a functional node based on the node attributes of the candidate node.

[0012] In a possible implementation, based on the second type of node features in the node features, to determine a functional node from candidate nodes, it includes: if the structural feature of candidate node p indicates that candidate node p includes one or more child nodes, obtain the node features of the child nodes of candidate node p; the node features of the child nodes of node p include the node width of the child nodes and the node height of the child nodes; if there is at least one child node of candidate node p whose node width is different from the node width of candidate node p, and the node width and node height of at least one child node are both greater than a second threshold, determine candidate node p as a functional node; where candidate node p is any one of the candidate nodes in the page to be processed.

[0013] It can be seen that when a candidate node includes one or more child nodes, it is possible to determine whether the candidate is a functional node based on features such as the node width of the candidate node, the node width of the child nodes, and the node height.

[0014] In a possible implementation, the non-functional nodes in the page to be processed include the child nodes of candidate node p, and the non-functional nodes are the nodes in the page to be processed except for the functional nodes.

[0015] It can be seen that after determining a node as a functional node, it is possible to determine that the child nodes included in the functional node are all non-functional nodes, so as to improve the efficiency of functional node recognition.

[0016] In a possible implementation, the above method may further include: using the preset display effect image of the page to be processed as the background image, marking the display area corresponding to the functional node in the background image to obtain a marked image; the ratio of the marked image is the same as the ratio of the background image; output the marked image in the image display interface according to the ratio of the marked image.

[0017] It can be seen that identifying and displaying the identified functional nodes in the background image can intuitively show the recognition result of functional node recognition, which is beneficial to adjusting the rules for identifying functional nodes based on the marked image and improving the accuracy of functional node recognition.

[0018] In a possible implementation, in the preset display area, the display area corresponding to the functional node includes the functional node and N child nodes of the functional node; N is an integer greater than or equal to 0.

[0019] It can be seen that the display area corresponding to the function node determined in this application in the preset display area may include the node itself determined as the function node and the N sub-nodes included in the function node, which is conducive to simultaneously planning the display effects of the function node and the sub-nodes of the function node in the preset display area.

[0020] In a second aspect, an embodiment of the present application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, and the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method described in the first aspect or any one implementation manner of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a chip system, which includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected by lines, and program instructions are stored in at least one memory; when the program instructions are executed by the processor, the chip system executes the method described in the first aspect or any one implementation manner of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer program product containing instructions. When the computer program product runs on an electronic device, the electronic device is caused to execute the method described in the first aspect or any one implementation manner of the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions. When the instructions run on an electronic device, the electronic device is caused to execute the method described in the first aspect or any one implementation manner of the first aspect. Description of the Drawings

[0024] Figure 1A is a schematic diagram of an application interface provided by an embodiment of the present application;

[0025] Figure 1B is a schematic diagram of another application interface provided by an embodiment of the present application;

[0026] Figure 1C is a schematic diagram of yet another application interface provided by an embodiment of the present application;

[0027] Figure 1D is a schematic diagram of yet another application interface provided by an embodiment of the present application;

[0028] Figure 2 is a schematic flowchart of formulating a page layout strategy provided by an embodiment of the present application;

[0029] Figure 3It is a schematic structural diagram of an AI processing system provided by an embodiment of the present application;

[0030] Figure 4 It is a schematic flowchart of a page data processing method provided by an embodiment of the present application;

[0031] Figure 5A It is a schematic diagram of an input file of a page provided by an embodiment of the present application;

[0032] Figure 5B It is a schematic structural diagram of a view tree provided by an embodiment of the present application;

[0033] Figure 6 It is a schematic flowchart of a method for determining functional nodes provided by an embodiment of the present application;

[0034] Figure 7A It is a schematic layout diagram of a page provided by an embodiment of the present application;

[0035] Figure 7B It is a schematic structural diagram of a view tree provided by an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of a functional node file provided by an embodiment of the present application;

[0037] Figure 9 It is a schematic diagram of a marked image corresponding to a page to be processed provided by an embodiment of the present application;

[0038] Figure 10 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application;

[0039] Figure 11 It is a schematic software structure diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0040] Next, the embodiments of the present application will be described in conjunction with the accompanying drawings in the embodiments of the present application.

[0041] Terms such as "first", "second", "third", and "fourth" in the specification, claims, and accompanying drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes unlisted steps or units, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0042] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one)" or its similar expression below refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0043] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0044] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, an application running on a computing device and the computing device can both be components. One or more components can reside in a process and / or an execution thread, and the components can be located on one computer and / or distributed between two or more computers. In addition, these components can execute from various computer-readable media on which various data structures are stored. The components can communicate, for example, through local and / or remote processes according to a signal having one or more data packets (such as data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems through a signal).

[0045] The user interface (UI) generally refers to the operation interface provided by a display device to a user. Display elements such as icons and text can be displayed in the user interface. There can be a certain interaction method between the display device and the user. For example, the display device can respond to operations such as clicks, touches, and swipes on the display elements to perform display or information transmission. For example, the display device can transmit an instruction to a player in an electronic device to play music corresponding to the instruction. Among them, the electronic device can include a mobile phone, a tablet computer, etc., and the display device can include a mobile phone screen, a tablet computer screen, etc.

[0046] The user interface can include the user interface in an application (abbreviated as an app), the operation interface of an intelligent driving device, the home screen interface of a mobile phone, etc. In the embodiments of this application, the user interface in an application is taken as an example for illustration.

[0047] The user interface (abbreviated as a page) corresponding to any application can include one or more of multiple pages such as a loading page, a guide page, a home page, and a content page. Usually, an app icon of the application can be displayed on the electronic device. When the electronic device detects a trigger operation (such as a click) on the app icon and responds to the trigger operation, the loading page of the application can be displayed. Among them, the loading page refers to the first page displayed after opening the application. Usually, the loading page can be composed of several parts such as a solid color or gradient background, a trademark (logo), and copyright information. The loading page is usually only displayed for a short period of time, such as 2 - 3 seconds. After the loading page is loaded, the guide page is usually displayed. The guide page generally includes several continuously displayed pages with a unified style, which are used to display the main functions and features of the application before it is officially used, or to introduce the new functions of the new version after the application is updated. The guide page is usually displayed when the application is first opened after being downloaded. If the application is not opened for the first time, after the loading page is loaded, the home page of the application can be directly displayed. The home page of the application is usually the most important page in the application, and some or all of the functions that the application can achieve can be displayed on the home page. For example, assuming the application is a shopping application, its home page can be as Figure 1A shown. The home page of the shopping application can be composed of the content of several parts such as a search bar at the top, multiple horizontal product bars, multiple vertical product bars, and a navigation bar at the bottom. Among them, the horizontal and vertical directions can be as Figure 1A indicated by the markings in. For example, the multiple horizontal product bars can be composed of 8 components such as a "recharge center" and a "platform subsidy", and 4 components such as "flash sale" and "live broadcast"; the multiple vertical product bars can be composed of components such as pictures and text information (such as prices) of multiple products. The page of the application displayed in response to the trigger operation on the home page can be called a content page. Usually, the content page can include various different functional types such as a list page, a detail page, a personal center page, a settings page, and a chat page. For example, when it is detected that forFigure 1A A trigger operation of the icon of any product in the multiple product columns shown can display a content page of the detail page function type in response to the trigger operation. The content page can be used to display the detailed information of the product, such as the product title, product price, product detail picture, monthly sales volume and other information.

[0048] Different electronic devices have different display parameters such as screen size, display ratio (such as aspect ratio) used to constrain the display effect of the page; and for the same electronic device, such as a foldable screen mobile phone, the screen size is different in the folded state and the unfolded state, resulting in different display effects of the same application page in the two screen states of the same electronic device. For example, a page can display complete information in the folded state of the mobile phone interface, but when the mobile phone is unfolded, the page may have a problem of reduced display information in the unfolded state of the mobile phone interface.

[0049] The folding methods of foldable screen mobile phones include folding inside out, folding up and down, etc.

[0050] For example, if the electronic device is a foldable screen mobile phone that folds inward and outward, and is in the folded state, the homepage of the shopping application page displayed on the foldable screen mobile phone may be as follows: Figure 1A When the foldable screen phone switches to the unfolded state, the screen size increases, and accordingly, the size of each display element on the page also increases based on a certain ratio. For example, suppose Figure 1A The size of the display element "Recharge Center" shown is 50 pixels × 50 pixels. In the expanded screen, Figure 1B As shown in FIG. 1 , the size of the “recharge center” can be increased to 80 pixels × 80 pixels. Figure 1B The size of each display element shown in the figure has also increased to a certain extent. Since the width and height (also called length) of each display element in the page are increased in the expanded state, while the height of the screen in the expanded state does not change, only the width changes, the screen in the expanded state may not be enough to display all the enlarged display elements, resulting in the lack of some content on the page in the expanded state, such as Figure 1B As shown, multiple vertical product columns may be missing, resulting in less display content on the homepage. If the size of each display element on the page remains unchanged after the foldable screen phone is switched to the unfolded state, more blank areas will appear on the left and right sides of the page, wasting display space.

[0051] If the electronic device is a foldable screen mobile phone that is folded in half and in an unfolded state, the following may be displayed: Figure 1A When a foldable screen phone switches to the folded state, the height of the phone screen decreases, while the width remains unchanged, thereby reducing the screen size. In one case, the size of the displayed elements in the page can remain unchanged. If the screen size is smaller than the original size, the screen size will be smaller than the original size.Figure 1A The sizes of the display elements shown in the figure remain unchanged. Due to the limitation that the screen size in the folded state is smaller than that in the unfolded state, the screen cannot display the following Figure 1A The complete content of the page shown results in a reduction in the amount of information on the application page, such as Figure 1C As shown, the page displayed in the folded state lacks multiple vertical product columns.

[0052] For example, if the electronic device is a tablet computer, the display effect of the shopping application page on the mobile phone can be as follows: Figure 1A The same shopping app page can be displayed on a tablet as shown in Figure 1D As shown in the figure. Since the screen size of a mobile phone is smaller than that of a tablet computer, if the size of each display element in the page remains unchanged, the page displayed on the tablet computer may include the following: Figure 1D However, there are many blank areas on the left and right sides of the tablet computer, which reduces the displayed content of the homepage and wastes a lot of display space.

[0053] Normally, for any application that needs to be displayed on an electronic device, after the application data of the application has been successfully downloaded to the electronic device, in order to ensure that the same page can display consistent and complete page content on the display devices of different electronic devices, and to ensure that the same page can display consistent content in different states of the same folding screen mobile phone (such as folded state and unfolded state), and avoid loss of information, developers can formulate customized page layout strategies based on the display parameters of the application and different types of electronic devices (such as screen size, aspect ratio, etc.), so as to ensure that the display effect of the electronic device can be adapted to the application page. For example, the same content can be displayed in both the folded and unfolded states of the folding screen mobile phone, and the amount of information contained is the same, thereby improving the user experience.

[0054] For example, see Figure 2 , Figure 2 A schematic diagram of the process of formulating page layout strategies for each application is shown. Figure 2 As shown, first, for each application, the developer can analyze and understand the structure of the page view, the size of the view and other information based on the page data of the application; based on the information obtained from the analysis, the developer can adjust the page layout strategy of each application, such as determining the position and size of each display element in the page, and the developer can debug and adjust the page layout strategy to obtain the target page layout strategy; after completing the setting and debugging of the page layout strategy, the configuration can be issued, that is, the page layout strategy corresponding to each application can be configured in the electronic device, so that the electronic device can display the page corresponding to each application based on the page layout strategy.

[0055] Among them, view is an object drawn on the screen for interacting with users. Usually, a page of an application includes multiple views, and each view can represent the same or different content. For example, as Figure 1A shown, the top navigation bar can be composed of one view; for another example, Figure 1A shown, multiple horizontal product bars can form one view, and "Recharge Center", "Platform Subsidy", "Online Supermarket", etc. can also be views. The structure in a view can refer to the structural relationship between each view; for example, the structural relationship can mean that one view is a child element or a parent element of another view; for example, as Figure 1A shown, the bottom navigation bar is the parent element of "Message" in the bottom navigation bar; "Message" is the child element of the bottom navigation bar. The size of a view can be used to indicate the size of a rectangular area occupied by the view on the screen, and the unit of the size can be pixels. For example, the size of the bottom navigation bar can be 800×200, indicating that the bottom navigation bar occupies a rectangular area on the screen with a width of 800 pixels and a height of 200 pixels.

[0056] Usually, electronic devices such as smart phones and tablet computers lack management tools such as a third-party access engine for constraining and managing the pages of third-party applications, making it difficult to require third-party applications to input application data into the electronic device according to a unified standard, and it is also difficult to require third-party applications to independently set and input various constraint conditions for the electronic device. Therefore, developers need to analyze each application separately, identify the constraint conditions of each application, and formulate matching layout strategies based on the constraint conditions of each application. This method is cumbersome, has limited generality, and has low engineering feasibility. If the electronic device is a folding-screen mobile phone, even for the same application, the electronic device needs to formulate two different layout strategies for the folded state and the unfolded state. Moreover, when the application is updated, developers also need to maintain it, resulting in too much human input and low practicality.

[0057] Based on this, the embodiments of the present application provide a page data processing method, which can be applied to an electronic device, enabling the electronic device to identify functional nodes in the page of an application based on artificial intelligence (AI) technology, which is beneficial to subsequent configuration of the page layout strategy based on the functional nodes.

[0058] Please refer to Figure 3 , Figure 3Schematic diagram of the structure of an AI processing system provided by an embodiment of the present application. The AI processing system may include multiple modules such as a data understanding module, a scene recognition module, a planning algorithm module, and a decision-making strategy module. The AI processing system may receive input data, which may include device information of an electronic device, application data, constraint conditions, and other data. Among them, the application data may include data in the page corresponding to the application, such as text, images, and other data in the page. The constraint conditions are used to describe the mutual positional relationship between each view in the page. During the process of the electronic device displaying the page, it can solve based on the constraint conditions in real time to determine the layout scheme in the current display area on the screen.

[0059] The data understanding module can be used to analyze the input data and determine information such as position information, relationship structure, and element characteristics from the input data. The position information may refer to the position information of the electronic device; the relationship structure may be the data structure in the application data, etc.; the element characteristics may be the data characteristics in the application data, such as data attributes, data storage methods, and other characteristics.

[0060] The planning algorithm module can be used to determine constraint conditions, perform high-performance solving, and determine evaluation criteria based on the processing results of the data understanding module. Among them, the constraint conditions are used to describe the mutual positional relationship between each view in the page. High-performance solving means that during the process of the electronic device displaying the page, it can solve based on the constraint conditions in real time to determine the layout scheme in the current display area on the screen. The evaluation criteria refer to the evaluation conditions used to evaluate the layout scheme in the current display area.

[0061] The scene recognition module can be used to perform module recognition and functional scene recognition on each application. Among them, based on the processing results of the data understanding module and the recognition algorithm determined by the planning algorithm module, the module recognition can identify each functional node in the application; the functional scene recognition refers to based on each identified functional node, identifying the specific functional scene corresponding to each functional node, such as various functional scenes such as a search bar, a vertical list, and a waterfall flow.

[0062] The decision-making strategy module can be used to perform zoom adjustment and add candidate elements to the display interfaces corresponding to each application based on the processing results of the planning algorithm module, and determine the layout strategy of each application based on the original strategy in the electronic device.

[0063] The AI processing system can output information such as the layout strategy and position information of each application obtained after being processed by the above multiple modules to the processor in the electronic device. Based on the layout strategy, position information, etc. of each application, the processor can perform rendering and drawing, so as to obtain a display interface adapted to each application.

[0064] Based on such as Figure 3The AI processing system shown can utilize AI technology to adaptively analyze the data of the page corresponding to the application, and adaptively perform function node recognition. While improving the accuracy and generality of function node recognition, it can significantly reduce the workload of developers.

[0065] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of a page data processing method provided by an embodiment of the present application. This method can be executed by an electronic device, or by a processor in the electronic device, or can also be executed by a chip or chip system with processor functions, etc. As Figure 4 shown, the page data processing method may include but is not limited to the following steps:

[0066] S401, obtain an input file of a to-be-processed page of a to-be-processed application.

[0067] The to-be-processed application can be any type of application. For example, the to-be-processed application can be a shopping application, a map application, an audio and video playback application, etc., and the present application does not make any limitations in this regard. For example, the to-be-processed application can be a shopping application as shown in Figure 1A . Optionally, after an electronic device finishes downloading an application, it can regard the application as a to-be-processed application and generate a download completion message for indicating to analyze and understand the page of the to-be-processed application. In response to the application download completion message, the electronic device can obtain an input file of a to-be-processed page of the to-be-processed application. Optionally, when an existing application in the electronic device completes a version update, it can regard the application as a to-be-processed application and generate a version update completion message to instruct the electronic device to obtain an input file of a to-be-processed page of the application.

[0068] The to-be-processed page of the to-be-processed application may include multiple pages such as a loading page, a guiding page, a home page, and a content page. If the number of to-be-processed pages of the to-be-processed application is multiple, the electronic device can process each to-be-processed page in sequence. For example, it can be the home page of a shopping application as shown in Figure 1A .

[0069] The input file of the page to be processed includes the page data of the page to be processed. The electronic device can determine information such as the relationship structure, size, and position of each view in the page to be processed based on the input file of the page to be processed, so as to determine the page layout strategy for the page to be processed. The input file of the page to be processed can be an Extensible Markup Language (XML) file. The page layout of the page to be processed can form a tree structure, which can be called a view tree. The "leaves" or "branches" in the view tree can be defined as nodes, and a node can be used to represent a view. In other words, the XML file of the page to be processed can include all views included in the page to be processed, that is, the data of all nodes. A node can include multiple data fields, and each data field can represent an attribute of a page.

[0070] Exemplarily, the input file of the page to be processed can be as Figure 5A shown. Figure 5A The labels ①-④ in Figure 5A represent 4 nodes. As Figure 5A shown, the node ① represented by ① includes two child nodes, namely node ② and node ④. In other words, node ① is the parent node of node ② and node ④, and node ② and node ④ are both child nodes of node ①. In addition, node ② is the parent node of node ③, and node ③ is the child node of node ②. Among them, each node includes multiple data fields such as those included in node ①. For the sake of concise description, Figure 5A the data fields included in each node are not fully shown in

[0071] Exemplarily, Figure 5A the view tree corresponding to nodes ①-④ shown in Figure 5B can be as Figure 5B shown. In this view tree, the root node is node ①, and node ② and node ④ are child nodes of node ①. In the view tree, it can be reflected that node ② and node ④ are branches of node ①. Node ③ is the child node of node ②, and in the view tree, it can be reflected that node ③ is a branch of node ②.

[0072] Exemplarily, when the application to be processed is a shopping application such as Figure 1A shown, the input file of the page to be processed can include multiple data fields as shown in Table 1.

[0073] Table 1 Comparison Table of Data Field Meanings

[0074] Data field Meaning class Node attribute content-desc Content description text Text resource-id Resource identifier index Hierarchy bounds Range isImportant Important attribute isVisible Visible attribute package Package name checkable Check attribute clickable Click attribute scrollable Scroll attribute long-clickble Long click attribute focused Focused attribute password Password attribute selected Selected attribute

[0075] Among them, "class" is a tag name used to indicate node attributes. For example, "class = \"android.widget.FrameLayout\"" means that the node attribute of this node is the frame layout of an application plugin (also called a component) in the Android system. Another example, "class = \"android.widget.TextView\"" means that the node attribute of this node is of the text type, and the content included in this node is mainly text content, such as being used to introduce the detailed information of a product (such as the place of origin, production date, etc.). "class = \"android.widget.ImageView\"" means that the node attribute of this node is the image attribute, and the content included in this node is mainly an image. For example, it can be a detail image of a product.

[0076] "content-desc" is a short text description used to describe components in the application to be processed, such as buttons, images, and text fields, etc. It is usually used for accessibility services to help users identify and operate these components. For example, "content-desc = \"This is a button\"" can indicate that the component formed by this node is a button. Another example, "content-desc = \"null\"" means that this node has no text description.

[0077] "text" represents the text content included in this node. For example, "text = \"Music\"" can indicate that the text "Music" is displayed at the position of this node in the page to be processed; "text = \"null\"" can indicate that no text content is displayed at the position of this node.

[0078] "resource-id" represents the resource identifier, which can be used to locate the node. Based on the "resource-id" of the node, the position of this node can be found in the input file of the page to be processed, and thus the data fields of this node can be obtained. For example, "resource-id = \"android:id / content\"" means that by inputting "android:id / content", the position of this node can be found in the input file of the page to be processed. "resource-id = \"null\"" means that the resource identifier of this node is empty. Among them, the resource identifiers of different nodes may be the same or different. If there are multiple nodes with the same resource identifier, assuming they are all \"android:id / content\", then by searching for \"android:id / content\" in the input file, all nodes with the resource identifier of \"android:id / content\" can be filtered out.

[0079] The "index" represents the level, that is, the level of this node, which can be used to represent the distance of this node from the root node in the view tree. The levels of different nodes may be the same or different. For example, different levels of two nodes mean that the levels of these two nodes are different; the same levels of two nodes mean that these two nodes are at the same level. Based on the "index" of multiple nodes, the structural relationship between each node can be determined. For example, "index" = "0" indicates that this node is the largest view, such as Figure 5A the node ① in Figure 5A and the levels of node ② and node ④ in Figure 5A are both 1, indicating that these two nodes are both child nodes of node ① and the distance from the root node is 1.

[0080] The "bounds" represents the range, which is used to indicate the coordinate and size information of the view represented by this node. For example, "bounds" = "[0, 0][1122, 91]" indicates that the coordinates (starting coordinates) of the upper-left pixel of this node are [0, 0], and the coordinates (ending coordinates) of the lower-right pixel are [1122, 91]. The rectangular area included by the upper-left pixel and the lower-right pixel is the range of this node. Based on this, the size (width × height) of the node can also be determined to be 1122 pixels × 91 pixels. Among them, the "bounds" of a specific node can be used to represent the screen size of the electronic device, such as Figure 5A the "bounds" = "[0, 0][1122, 2442]" in

[0081] "isImportant" is used to indicate whether this node is important, and this attribute can be set in the application to be processed. For example, "isImportant" = "false" can indicate that this node is marked as unimportant; "isImportant" = "true" can indicate that this node is marked as important. Among them, whether a node is "important" can be set and modified, and different processing methods can be adopted for "important" nodes and "unimportant" nodes. This application does not limit the specific processing methods.

[0082] "isVisible" is used to indicate whether this node is a visible node. For example, "isVisible" = "true" can indicate that this node is a visible node and can be displayed in the electronic device for the user to directly view, such as Figure 1AThe navigation bar at the top shown. isVisible = "false" indicates that this node is an invisible node, such as the hidden border line of the view.

[0083] The other data fields in Table 1 and their corresponding meanings are as described in Table 1, which will not be elaborated here.

[0084] In a possible implementation, the input file of the page to be processed can be obtained based on the accessibility service in the Android system. Exemplarily, based on the accessibility service, node extraction can be performed on the original data of the page to be processed, multiple nodes required by the electronic device can be extracted, and the extracted nodes can be filtered to retain multiple data fields as shown in Table 1, so as to determine the data fields included in each node. Each node is arranged and stored according to the structure of the view tree, and an XML file can be formed.

[0085] S402, based on the input file, determine the node features of each node in the page to be processed.

[0086] The node features of each node in the page to be processed refer to the feature information related to the page layout. Some node features of the node can be directly obtained from the data fields of this node in the input file, and the other part of the node features can be determined based on the data fields of one or more nodes. The node features of each node can be divided into two categories. The first category can be size information, which can be called the first type of node features; the second category can be view attribute information, which can be called the second type of node features.

[0087] Among them, the size information, that is, the first type of node features can include information such as the node width and node height of the node. The size information can be determined from the data field bounds as shown in Table 1 above. For example, if the data field bounds of a node is "[10, 20][500, 780]", it can be determined that the node width of this node is 490 pixels and the node height is 760 pixels.

[0088] The view attribute information, that is, the second type of node features can include the structural relationship information and view attributes of the view. The view attributes can include the feature information represented by data fields such as the node attribute (class) and level (index) in Table 1. Based on the node attribute, it can be determined whether the content of the node is text type or picture type; based on the index, the level of the node can be determined. The structural relationship of the view can also be called the structural feature of the node, which is used to represent the structural relationship between nodes, such as the number of child nodes included in the node. Through the level of a node and the levels of multiple adjacent nodes to this node, the structural feature of the node can be determined. For example, it can be determined as Figure 5ANode ① in it is the parent node of Node ② and Node ④. If a node includes one or more child nodes, the feature information of the node also needs to include the node width of the child nodes.

[0089] In summary, the node features of a node can include node width, node height, node attributes, hierarchy, and structural features, and the node width of child nodes. Optionally, the node features of a node can include some or all of the above features, and can also include other features other than the above node features, such as the sliding attribute, concentration attribute, etc. shown in Table 1. This application does not make any limitations on this.

[0090] S403, based on the node features of each node, determine the nodes in the page to be processed that meet the preset node feature conditions as functional nodes.

[0091] A functional node is a node in the page to be processed, and can also be called a functional module, display module, etc. A functional node occupies a certain area in the page to be processed and is used to display the page content represented by the data fields of the functional node. A functional node can include one or more child nodes, that is, a functional node can include one or more child nodes. In other words, a functional node can include one or more views.

[0092] Usually, a page to be processed can include multiple functional nodes. For example, Figure 1A In the home page of the shopping application shown, the search bar at the top, the navigation bar at the bottom, etc. can all be functional nodes.

[0093] Based on the node features of the node (including the first type of node features and the second type of node features), it can be determined whether the node is a functional node. The judgment condition for determining whether a node is a functional node can be called a preset node feature condition. The preset node feature condition can be configured in the electronic device. Optionally, the preset node feature condition can also be called a preset node recognition rule. Traverse all the nodes included in the input file of the page to be processed. Based on the preset node feature condition, it can be sequentially determined whether the feature information of each node meets the preset node feature condition, so as to determine whether the node is a functional node. If the node features of the node meet the preset node feature condition, it can be determined that the node is a functional node.

[0094] Among them, based on the first type of node features and the second type of node features of the node, the preset node feature condition can be divided into a first feature condition and a second feature condition. The first feature condition can include a height condition and a width condition; the second feature condition can include a node attribute condition or a child node condition. For the specific content of each condition in the preset node feature condition, please refer to the following description.

[0095] For any node (which can be called node i) in the page to be processed, the method for determining whether the node is a functional node can be as followsFigure 6 As shown. Exemplarily, as Figure 6 shown, taking node i in the page to be processed as an example, the method may include but is not limited to the following steps:

[0096] S601, based on the data field of node i in the input file, determine the node features of node i.

[0097] The first type of node features of node i may include features such as node width and node height, and the second type of node features may include one or more of structural features, node attributes, hierarchy, etc. If node i includes one or more child nodes, the feature information of node i may further include the node features of all child nodes of node i, such as features of the child node's node width, node height, node attributes, etc.

[0098] S602, determine whether the node width of node i meets the width condition. If the node width of node i meets the width condition, execute S603; otherwise, execute S608.

[0099] Based on the data field bounds of node i, the node width of this node can be determined. For example, if bonus = "[0,0][216,890]", it can be determined that the node width is 216 pixels. In addition, from the data field bounds included in the first node in the input file of the page to be processed, the width of the electronic device (referring to the screen width of the electronic device) can be determined; or, the screen width of the electronic device can be directly obtained from the electronic device. Exemplarily, hereinafter, use w1 to represent the node width and w0 to represent the screen width. Both w1 and w0 are integers greater than 0, and the units of w1 and w0 are both pixels.

[0100] The width condition is the width condition in the preset node feature conditions. If the node width of node i meets any one of the following width conditions A - C, it can be determined that this node meets the width condition:

[0101] A. The node width is equal to the screen width, which can be expressed as: w1 = w0;

[0102] B. The node width is equal to half of the screen width, which can be expressed as:

[0103] C. The node width is greater than or equal to three - quarters of the screen width, which can be expressed as:

[0104] Among them, if node i meets the width condition A, it means that the node is a node with the same width as the screen width. Usually, the node that meets the width condition A can be the largest functional node in the guiding page, such as a functional node used as the background. If the node meets the width condition B, it can indicate that the node may be a module occupying a relatively large display area on the screen. For example, it can be multiple vertical product columns as shown in Figure 1A shown. If the node meets the width condition C, it can indicate that the node is a relatively wide functional node such as the search bar at the top or the navigation bar at the bottom as shown in Figure 1A shown.

[0105] S603. Determine whether the node height of node i meets the height condition. If the node width of node i meets the height condition, execute S604; otherwise, execute S608.

[0106] The height condition is the height condition in the preset node feature conditions, which can be expressed as: the node height of node i is greater than the first threshold. The height of the node can be determined based on the data field bounds of node i. Optionally, the size of the first threshold can be set according to the actual application scenario. For example, the first threshold can be set to 2 pixels.

[0107] Exemplarily, if the abscissas of the starting coordinate and the ending coordinate of node i are equal, such as bounds = "[10, 210][110, 210]", based on this, the node height of this node can be determined to be 0, indicating that the node may be a horizontal dividing line. Or, if the ordinates of the starting coordinate and the ending coordinate of node i are equal, such as bounds = "[10, 210][10, 400]", based on this, the node width of this node can be determined to be 0, indicating that the node may be a vertical dividing line.

[0108] Among them, if the node width of node i meets the width condition and the node height of node i meets the height condition, then it can be said that the node feature of node i meets the first feature condition. Optionally, if the node feature of node i meets the first feature condition, this node can be determined as a candidate node. In other words, when node i meets the first feature condition, it can be excluded that node i is a dividing line or a node with too small corresponding display area, which is beneficial to the accuracy of determining the functional node from the candidate nodes subsequently.

[0109] Furthermore, it is also necessary to determine whether the candidate node is a functional node based on whether the second type of node feature of the node meets the second feature condition.

[0110] Exemplarily, assume a page to be processed as shown in Figure 7A shown, where the labels a - f respectively represent 6 views. In other words, it can represent 6 nodes. Node d can include one or more sub - nodes, and node f can also include one or more sub - nodes. Figure 7AIn the figure, the sub-nodes of node d are represented by small squares, and the sub-nodes of node f are represented by large squares. For Figure 7A any one of the nodes in, based on its first type of node features, it can be determined whether each node is a candidate node. For example, the node width of node a is equal to the screen width, so node a is not a candidate node. The node widths of nodes b, c, d, and f are all greater than or equal to three-quarters of the screen width, and the node heights are greater than the first threshold (assumed to be 0 pixels). Therefore, it can be determined that nodes c, d, and f are also candidate nodes. The node widths of the sub-nodes included in nodes d and f do not meet the width condition, so neither of them is a candidate node. The node width of node e is greater than or equal to three-quarters of the screen width, but the node height is 0. Therefore, node e is not a candidate node.

[0111] In a possible implementation, for node i, it can first be determined whether the node height of node i meets the height condition; when the node height of node i meets the height condition, it is determined whether the node width of node i meets the width condition. Based on this, when the node height of node i meets the height condition and the node width meets the width condition, it can be determined that node i meets the first feature condition, and node i can be determined as a candidate node.

[0112] S604, determine whether node i is a single node. If node i is a single node, then execute S605; otherwise, execute S606.

[0113] Based on the structural features of the node, it can be determined whether the node is a single node. If the structural features of a node indicate that the node has no sub-nodes, then the node can be called a single node; correspondingly, if the structural features of a node indicate that the node includes one or more sub-nodes, then the node is not a single node and can be called a non-single node. Based on the data field index, the level of node i can be determined. Based on the levels of other nodes adjacent to node i in the input file of the page to be processed, it can be determined whether node i has a parent node or sub-nodes, and thus the structural features of node i can be determined. For example, based on the levels of node ① and nodes ② - ④ adjacent to node ① in Figure 5A , it can be determined that node ① is the parent node of nodes ② and ④. Therefore, node ① is not a single node and can be called that the structural features of node ① indicate that node ① includes 2 sub-nodes. As in Figure 5A , nodes ③ and ④ have no sub-nodes. Therefore, both nodes ③ and ④ are single nodes, and it can be called that the structural features of node ③ indicate that node ③ has no sub-nodes, and the structural features of node ④ indicate that node ④ has no sub-nodes.

[0114] In addition, if the node width of all child nodes included in node i is less than the first threshold, or the node height is less than the first threshold, node i can also be regarded as a single node, and thus S605 is executed. Among them, the node width and node height of the child nodes of node i can be determined based on the bounds of each child node.

[0115] S605. Determine whether the node attribute of node i is a text attribute. If the node attribute of node i is a text attribute, then execute S607; otherwise, execute S608.

[0116] Among them, if node i is a single node and the node attribute of node i is a text attribute, it can be said that node i meets the node attribute condition; if node i is a single node and the node attribute of node i is not a text attribute, it can be said that node i does not meet the node attribute condition.

[0117] Based on the value of the node attribute (class) of node i, it can be determined whether node i is a text attribute. For example, if class in the characteristic information of node i = "android.widget.TextView", it can indicate that the node attribute of node i is a text attribute. If class = "android.widget.ImageView", it can indicate that the node attribute of node i is a picture attribute.

[0118] In a possible implementation, the conditions that the node attribute needs to meet when the node is a single node can be adjusted based on different application scenarios. For example, if the application to be processed is a picture-based application, the condition in S605 can be modified to: node i is a single node, and the node attribute of this node is a picture attribute. In this way, the pictures in the picture-based application can also be regarded as a separate module.

[0119] S606. Determine whether the child nodes of node i meet the child node conditions. If the child nodes of node i meet the child node conditions, then execute S607; otherwise, execute S608.

[0120] The child node conditions are used to represent the conditions that the child nodes of a non-single node need to meet. When node i is not a single node, if the child nodes of child node i meet the following conditions, it can be determined that the child nodes of node i meet the child node conditions: there exists a child node whose node width is different from the node width of node i, and the node width and node height of this child node are both greater than the second threshold.

[0121] Among them, node i may include one or more child nodes, and the node width of each child node may be determined based on the data field bounds of each child node. Among the one or more child nodes included in node i, there may be one or more child nodes whose node widths are different from the node width of node i. The second threshold may be set to any pixel value. Usually, the second threshold may be equal to the first threshold, for example, both are set to 2 pixels. Usually, if the node width or node height of a child node is less than the second threshold, the child node may be regarded as a dividing line for separating different nodes.

[0122] Usually, when the node width or node height of a child node is less than the second threshold, it means that the display area occupied by the child node on the screen is small, and it is usually used as a horizontal dividing line or a vertical dividing line for separating different nodes. Usually, the dividing line does not include information such as text and pictures. If the node width or node height of each child node included in node i is less than the second threshold, it means that the child nodes in node i can all be regarded as dividing lines and do not contain information. Therefore, in this case, it can be determined that the child nodes of node i do not meet the child node conditions, and thus S608 is executed.

[0123] S607, determine that node i is a functional node and determine that the child nodes of node i are non-functional nodes.

[0124] In summary, when node i simultaneously meets the width condition and the height condition, and node i is a single node with text attributes, that is, the node characteristics of node i simultaneously meet the first characteristic condition (width condition and height condition) and the second characteristic condition (node attribute condition), it can be determined that node i is a functional node; or, when node i simultaneously meets the width condition and the height condition, and node i is a non-single node, and the child nodes of node i meet the child node conditions, that is, the node characteristics of node i simultaneously meet the first characteristic condition (width condition and height condition) and the second characteristic condition (child node condition), it can be determined that node i is a functional node. In addition, when it is determined that node i is a functional node, it can be determined that the child nodes of node i are non-functional nodes. That is, when traversing each node in the page to be processed, the child nodes of the nodes that have been determined to be functional nodes can be skipped.

[0125] For any functional node, pruning judgment is required. Among them, pruning judgment means that if a node has been determined to be a functional node, if the node includes other functional nodes that have been determined to be functional nodes currently, or the node width of the node is equal to the screen width of the electronic device, it is determined that the node meets the pruning condition, and thus the node can be re-determined as a non-functional node. In other words, pruning judgment is a process of checking and correction, used to check whether there are errors in the functional nodes determined based on the above first-category and second-category node characteristics and correct them in time.

[0126] Exemplarily, such as Figure 7AThe structure of the view tree of the input file of the page to be processed shown can be as Figure 7B shown. As Figure 7B shown, the root node of the view tree is node a, the child node of node a is node b; the child nodes of node b include nodes c - f. The multiple child nodes included in node d, and the multiple child nodes included in node f, Figure 7B are not drawn in. Based on the preset node feature conditions and the idea of the clustering algorithm based on the node hierarchy, according to the hierarchy of the nodes, traverse each node in the view tree as Figure 7B shown in turn, so as to determine the functional nodes. Assume that nodes c - f have been determined as candidate nodes among nodes a - f. For example, since node a has been determined not to be a candidate node, traversal can continue starting from node b. Node b is a single node, but the node attribute of node b is not a text attribute but a picture attribute, which is a background picture, so it can be determined that node b is not a functional node. Node c is a single node, and the node attribute is a text attribute, so it can be determined that node c is a functional node. Node d is not a single node, and the node widths of the child nodes of node d do not meet the width condition, so the child nodes included in node d are not functional nodes, and moreover, the node widths of the child nodes of node d are different from the node width of node d, so it can be determined that node d is a functional node and there is no need to continue traversing the child nodes included in node d. Node e is not a candidate node and can be skipped. Node f is similar to node d. The child nodes included in node f are not functional nodes, and the node widths of the child nodes are different from that of node f, so it can be determined that node f is a functional node and there is no need to continue traversing the child nodes included in node f. Based on this, it can be determined that nodes c, d, and f are functional nodes.

[0127] Optionally, the specific conditions in the preset node feature conditions can be adaptively modified, added, or deleted based on the application scenario, and the present application does not limit this.

[0128] In a possible implementation manner, optionally, first, based on the first type of node features, determine whether a node is a candidate node, such as the above steps S602 - S603; if it is a candidate node, then further, based on the second type of node features, determine whether this node (i.e., the candidate node) is a functional node, such as the above steps S604 - S606. If it is determined that a node is not a candidate node, then it can be determined that this node is not a functional node. Optionally, for any node, first, based on the second type of node features, determine whether this node is a candidate node. If this node is a candidate node, then based on the first type of node features, determine whether this candidate node is a functional node, that is, determine whether this node is a functional node.

[0129] S608, determine that node i is a non - functional node.

[0130] If node i does not meet the width condition or the width condition, or does not meet the node attribute condition when node i is a single node, or does not meet the child node condition when node i is a non-single node, it can be determined that node i does not meet the preset node feature condition, and thus it can be determined that node i is a non-functional node.

[0131] Based on the method as Figure 6 shown, each node in the page to be processed can be traversed, nodes that meet the preset node feature conditions can be determined, and these nodes can be determined as functional nodes.

[0132] S404. Based on the functional nodes, determine the functional node file corresponding to the page to be processed.

[0133] The functional node file includes some or all of the data fields of the functional nodes in the page to be processed. The functional node file can be used to adjust the page layout of the page to be processed. The functional node file can be a JS key-value pair data (JavaScript object notation, JSON) file. Exemplarily, the format of the functional node file can be as Figure 8 shown.

[0134] As Figure 8 shown, the functional node file can include the data fields of each functional node in the page to be processed, can include some or all of the data fields shown in Table 1, and the data fields determined based on the data fields shown in Table 1. For example, the functional node file can include the package name (package), node identity identifier (node_indexId), node identifier (node_id), level (index), node attribute (class), range (bounds), width (width), height (height), number of child nodes (son_nums), click attribute (clickable), scroll attribute (scrollable), and location attribute (loc) of each functional node in the page to be processed.

[0135] Among them, the node identity identifier (node_indexId) is the unique identity identifier of each node in the page to be processed. Based on the natural order of the arrangement of each node in the input file of the page to be processed, each node can correspond to a node identity identifier in turn. For example, in the input file as Figure 5A shown, in order, node ① is the first node in the page to be processed, then the node identity identifier of node ① can be expressed as node_indexId: "index: id / 1"; node ② - node ④ are the second, third, and fourth nodes in the page to be processed in turn. For example, the node identity identifier of node ③ can be expressed as node_indexId: "index: id / 3".

[0136] The node_id is the same data field as the resource-id in Table 1 above. The level (index), node attribute (class), and range (bounds) are the same data fields as the level (index), node attribute (class), and range (bounds) in Table 1 respectively.

[0137] The width and height are determined based on the bounds. The width refers to the node width of the node, and the height refers to the node height of the node.

[0138] The position attribute is a combined vector of the coordinates of the pixels at the upper left corner and the lower right corner of the functional node, which can be used to determine the position of the functional node. The position attribute can be determined based on the range of the node. For example, based on "bounds": "[36, 542, 1086, 1102]", "loc": "[36, 542, 1086, 1102]" can be determined.

[0139] Figure 8 Among them, the functional node with the functional node identifier of 25 can be called the main functional node in the page to be processed. The 9 functional nodes with the functional node identifiers of 26 - 34 are the sub-functional nodes of the main functional node. Each sub-functional node also includes the above-mentioned multiple data fields. For the sake of concise description, Figure 8 the data fields included in each sub-functional node are not shown in detail.

[0140] S405. Based on the functional node file and the preset display effect image of the page to be processed, determine the marked image corresponding to the page to be processed, and output the marked image on the image display interface.

[0141] The image display interface can refer to any interface in the electronic device that can be used to display images. The marked image corresponding to the page to be processed refers to an image obtained by marking the display area corresponding to the identified functional node in the background image with the preset display effect image corresponding to the page to be processed as the background image based on the functional node. The marked image corresponding to the page to be processed can also be called the image corresponding to the recognition result of the functional nodes of the page to be processed. Among them, the preset display effect image corresponding to the page to be processed refers to an image in which the preset page content is completely displayed on the screen of the page to be processed. Exemplarily, by performing operations such as taking a screen shot of the interface displayed by the electronic device in Figure 1A the like, the preset display effect image corresponding to the page to be processed can be obtained.

[0142] Optionally, the electronic device may output the marked image to the gallery in the electronic device and display the marked image in the image display interface corresponding to the gallery. Optionally, the electronic device may output the marked image to another electronic device, and the other electronic device receives the marked image and displays the marked image in the image display interface of the other electronic device.

[0143] Based on the marked image, developers can verify the accuracy of function node recognition and can also perform adaptive modification, adjustment, etc. on the preset node feature conditions, so as to ensure that each function node in the page to be processed can be accurately recognized based on the preset node feature conditions.

[0144] Exemplarily, the marked image displayed in the image display interface may be as Figure 9 shown. If the application to be processed is a shopping application, the following function nodes 1 - 5 can be recognized on the page to be processed as Figure 9 shown. Function node 1 can be called the top search bar and is used to receive search keywords. Function node 2 can be called the horizontal treasure box and may include multiple parts such as a "recharge center" and a "platform subsidy". Function node 3 can be called the horizontal product bar and may include multiple parts such as "flash sale" and "live broadcast". Function node 4 can be called the product waterfall flow and may include information such as introduction pictures and prices of multiple products. Function node 5 can be called the bottom navigation bar and may include multiple parts such as "home page" and "message".

[0145] In a possible implementation, based on the AI processing system as Figure 3 shown, the scene recognition module can determine the function scenes corresponding to each function node based on the function node file, such as multiple function scenes like search bars, vertical lists, waterfall flows, etc. For different function scenes, the decision-making strategy module can adjust the page to be processed based on the function node and the function scene, and in combination with the preset layout parameters of the preset display area used to display the page to be processed on the screen, such as parameters like the width, height, and resolution of the preset display area, so as to determine the layout strategy of the page to be processed. Among them, the preset display area can be the screen area in the unfolded state of the folding screen or the screen area in the folded state, or the outer screen area of the folding screen, etc.; the layout strategy may include the size and proportion of each node in the corresponding display area on the screen, the positional relationship between each node, etc. Further, the processor in the electronic device can render and draw the page to be processed based on the layout strategy determined by the AI processing system, so as to obtain the display page corresponding to the page to be processed adapted to the electronic device.

[0146] It can be seen that based on the formulated preset node feature conditions and the node features of each node in the page to be processed, the functional nodes in the page to be processed can be accurately identified, improving the accuracy and efficiency of the electronic device in understanding and recognizing the page to be processed, thereby facilitating the planning of the layout strategy of the application to be processed on different display screens. In addition, the embodiment of the present application can also display the recognition result of the functional node in a visual manner, that is, display the marked image corresponding to the page to be processed, which is beneficial for developers to adjust the preset node feature conditions to further improve the accuracy of functional node recognition.

[0147] The following introduces an exemplary electronic device provided by an embodiment of the present application. Please refer to Figure 10 , Figure 10 which shows a schematic structural diagram of the electronic device 100.

[0148] The following takes the electronic device 100 as an example to specifically illustrate the embodiment. It should be understood that the electronic device 100 may have more or fewer components than those shown in Figure 10 , may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0149] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0150] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0151] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0152] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0153] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0154] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0155] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are only illustrative and do not constitute a structural limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0156] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0157] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flexible light-emitting diode (FLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0158] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0159] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise and brightness of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be provided in the camera 193.

[0160] The camera 193 is used to capture still images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include other cameras.

[0161] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0162] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0163] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously learn on its own. Through the NPU, applications such as the intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0164] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0165] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0166] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0167] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0168] Figure 11 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0169] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer (framework), system runtime layer, hardware abstraction layer (HAL) and kernel layer (kernel).

[0170] The application layer can include a series of application packages.

[0171] like Figure 11As shown, the application package may include applications such as the system user interface (systemUI), camera, calendar, WLAN, music, short message, gallery, call, map, Bluetooth, video, etc. (which may also be referred to as apps), and an AI processing system as shown in Figure 3 Among them, the system UI is used to display the interface of the electronic device 100, such as displaying the signal icon corresponding to the SIM card, displaying the call interface, etc. In the embodiments of the present application, the system UI can be used to display each application interface as shown in Figure 1A - Figure 1D . The AI processing system can receive the input file of the page to be processed of any application in the application layer and determine the functional nodes in the page to be processed based on the input file.

[0172] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0173] Such as Figure 11 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0174] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0175] The content provider is used to store and obtain data and make this data accessible to applications. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc.

[0176] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0177] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including connection, disconnection, etc.).

[0178] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0179] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is completed, a message is reminded, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a graph or a scroll bar text, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialogue interface. For example, it can prompt text information in the status bar, emit a prompt sound, vibrate the electronic device, blink the indicator light, etc.

[0180] The system runtime layer is divided into two parts, namely the C / C++ library and the Android runtime library. The Android runtime includes the runtime environment, which usually includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The C / C++ library mainly includes the multimedia library (media framework), the surface manager, the 3D graphics processing library (e.g., OpenGL ES), the 2D (two-dimensional) graphics engine (e.g., SGL), etc.

[0181] Among them, the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, layer processing, etc. The 2D graphics engine is the drawing engine for 2D drawing. In the embodiments of the present application, the 3D graphics engine and the 2D graphics engine can be used to render and draw the interfaces of various applications.

[0182] The hardware abstraction layer is used to isolate the application framework layer from the kernel layer, avoiding the Android system's over-reliance on the kernel layer, so that the development of the application framework layer can be carried out without considering the driver. The hardware abstraction layer can include multiple functional nodes. For example, modules such as the display HAL, the camera HAL, the audio HAL, and the sensor HAL.

[0183] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes the display driver, the camera driver, the audio driver, the sensor driver, and the shared memory driver. In the embodiments of the present application, the display driver can be used to display the interface of the application.

[0184] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0185] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps may be implemented in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0186] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0187] The units described as separate components above may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, in each embodiment of this application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0189] When the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute all or part of the steps of the methods in the various embodiments of this application. Among them, the aforementioned storage medium can include: various media that can store program codes such as USB flash drives, mobile hard disks, magnetic disks, optical discs, read-only memory (ROM), or random access memory (RAM).

[0190] As described above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application.

Claims

1. A page data processing method, characterized in that, The method includes: When receiving the information that the download of an application is completed or the version update is completed, based on the page data of the page to be processed of the application, determining the node features of the nodes in the page to be processed; the node features include first - type node features and second - type node features; the first - type node features include the node width and node height of the node; the second - type node features include the structural features and node attributes of the node. Based on the first - type node features, determining candidate nodes from the nodes of the page to be processed. Based on the second - type node features, determining functional nodes from the candidate nodes. Based on the functional nodes, determining the functional node file corresponding to the page to be processed; the functional node file includes some or all of the data fields of the functional nodes. Based on the functional node file and preset layout parameters, adjusting the display size and display position of the functional nodes in a preset display area to determine the page layout corresponding to the page to be processed; the preset display area is the area on the screen for displaying the page to be processed; the preset layout parameters include the width and height of the preset display area.

2. The method according to claim 1, wherein The determining candidate nodes from the nodes of the page to be processed based on the first - type node features includes: Determining candidate nodes from the nodes of the page to be processed, where the node height is greater than a first threshold and the node width meets a first condition. Where candidate node i is any one of the candidate nodes, the node width of candidate node i meeting the first condition includes one or more of the following: The node width of candidate node i is equal to the screen width. The node width of candidate node i is equal to one - half of the screen width. The node width of candidate node i is greater than or equal to three - quarters of the screen width.

3. The method according to claim 1, wherein The determining functional nodes from the candidate nodes based on the second - type node features includes: If the structural features of candidate node p indicate that candidate node p has no child nodes and the node attribute of candidate node p is a text attribute, determining candidate node p as a functional node; candidate node p is any one of the candidate nodes in the page to be processed.

4. The method according to claim 1, characterized in that, The determining functional nodes from the candidate nodes based on the second - type node features includes: If the structural features of candidate node p indicate that candidate node p includes one or more child nodes, obtaining the node features of the child nodes of candidate node p; the node features of the child nodes of node p include the node width and node height of the child nodes. If there is at least one child node of candidate node p whose node width is different from the node width of candidate node p and the node width and node height of the at least one child node are both greater than a second threshold, determining candidate node p as a functional node. Where candidate node p is any one of the candidate nodes in the page to be processed.

5. The method according to claim 4, characterized in that, The non - functional nodes in the page to be processed include the child nodes of candidate node p, and the non - functional nodes are the nodes in the page to be processed other than the functional nodes.

6. The method according to any one of claims 1-5, characterized in that, In the preset display area, the display area corresponding to the function node includes the function node and N sub-nodes of the function node; N is an integer greater than or equal to 0.

7. The method according to any one of claims 1 to 5, characterized in that The method further includes: Using the preset display effect image of the page to be processed as the background image, marking the display area corresponding to the function node in the background image to obtain a marked image; the ratio of the marked image is the same as the ratio of the background image; Outputting the marked image on the image display interface according to the ratio of the marked image.

8. An electronic device, characterized in that, It includes: A memory and a processor; wherein: The memory is used to store a computer program, and the computer program includes program instructions; The processor is used to call the program instructions to cause the electronic device to execute the method according to any one of claims 1-7.

9. A chip system, characterized in that, The chip system includes at least one processor, a memory and an interface circuit. The memory, the interface circuit and the at least one processor are interconnected by lines. Program instructions are stored in the at least one memory. When the program instructions are executed by the processor, the chip system executes the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1-7 is implemented.

Images