Interface component identification method and device, equipment and storage medium

Abstract

Embodiments of the invention provide an interface component identification method and apparatus, a device and a storage medium, which are used for improving the accuracy of component identification in an interface. Comprising the steps that control tree information of a current interface is obtained, and the control tree information is used for indicating rendering priorities of multiple canvases of the current interface; determining a first rendering priority of the plurality of canvases according to the control tree information; traversing the canvases according to the sequence of the first rendering priorities from high to low, obtaining the area of the canvases, and calculating the ratio of the area of the canvases to the area of a screen, the area of the screen being the display area of a display device displaying the current interface; when the ratio of the canvas area to the screen area meets a first preset condition, stopping traversing the canvas, and taking the current canvas as a target canvas; and obtaining at least one target component on the target canvas. The technical scheme provided by the invention can be applied to the fields of artificial intelligence, cloud technologies, computers and the like.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an identification method, device, equipment and storage medium for an interface component. Background Art

[0002] Under the tide of the Internet, games or other corresponding applications are increasingly inclined to global development. In this process, since the language used by different countries when developing games or applications is generally the native language of that country, in order to meet the needs of the global development of games or applications, the language that appears in the game can generally be translated into other languages ​​so that the language text of the corresponding language can be displayed in the game.

[0003] In general, in game development testing, in order to ensure the translation quality of the language text in the game, localization quality assurance (LQA) is required. The existing LQA needs to retrieve screenshots of the language text to be checked in the game or application, as well as screenshots of the translated language text in different languages ​​corresponding to the language text to be checked, for comparison. At present, each component in the game interface or application interface is usually identified, and then the screenshots are reported. However, when identifying components, it is possible that unnecessary underlying components are also identified, making component identification inaccurate.

[0004] Therefore, how to quickly and accurately identify each component in the game interface is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide an interface component recognition method, apparatus, device, and storage medium, which are used to provide an accurate component recognition method, thereby improving the accuracy of local testing.

[0006] In view of this, on the one hand, the present application provides a method for identifying interface components, including: obtaining control tree information of the current interface, the control tree information is used to indicate the rendering priority of multiple canvases of the current interface, and the canvas is used to carry and render each component of the current interface; determining a first rendering priority of the multiple canvases according to the control tree information; traversing each canvas in order from high to low according to the first rendering priority, obtaining the canvas area of ​​the canvas and calculating the ratio of the canvas area to the screen area, the screen area being the display area of ​​a display device displaying the current interface; when the ratio of the canvas area to the screen area meets a first preset condition, stopping traversing the canvas and taking the current canvas as the target canvas; and obtaining at least one target component on the target canvas.

[0007] On the other hand, the present application provides an interface component identification device, comprising: an acquisition module, used to acquire control tree information of a current interface, the control tree information is used to indicate the rendering priority of multiple canvases of the current interface, the canvas is used to carry and render various components of the current interface;

[0008] A processing module is used to determine a first rendering priority of the multiple canvases according to the control tree information; traverse each canvas in order from high to low according to the first rendering priority, obtain the canvas area of ​​the canvas and calculate the ratio of the canvas area to the screen area, where the screen area is the display area of ​​the display device that displays the current interface; when the ratio of the canvas area to the screen area meets a first preset condition, stop traversing the canvas and use the current canvas as a target canvas; and obtain at least one target component on the target canvas.

[0009] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to obtain a component information set on the target canvas, where the component information set is used to indicate rendering priorities of multiple components of the target canvas;

[0010] Determine a second rendering priority of the multiple components according to the component information set;

[0011] Traverse each component in descending order of the second rendering priority to obtain the coverage area of ​​the component;

[0012] When there is a first component whose coverage area and the screen area have a ratio that meets the second preset condition, the first component and the component before the first component are taken as the at least one target component.

[0013] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to obtain a component information set on the target canvas, where the component information set is used to indicate rendering priorities of multiple components of the target canvas;

[0014] Determine a second rendering priority of the multiple components according to the component information set;

[0015] Traverse each component in descending order of the second rendering priority, and obtain the coverage area of ​​the components in sequence;

[0016] When there is a second component whose coverage area and the screen area have a ratio that satisfies a second preset condition, the second component and the components before the second component are taken as an intermediate component set;

[0017] Rectangular collision detection and rectangular coverage detection are performed on each component in the intermediate component set to filter out the blocked parent component, so as to obtain the at least one target component, which is the parent component in the intermediate component set.

[0018] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to perform rectangular collision detection and rectangular coverage detection on a first group of components in the intermediate component set to obtain a detection result, the first group of components includes a third component and a fourth component, and the first components are obtained by grouping in pairs;

[0019] When the detection result determines that there is a rectangular collision or rectangular overlap between the third component and the fourth component, determine the parent component and child component relationship between the third component and the fourth component, filter out the parent components of the third component and the fourth component, and use the child components of the third component and the fourth component as the target component;

[0020] Repeat the above operation until the set of intermediate components is traversed and the at least one target component is obtained.

[0021] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to obtain a first upper left corner coordinate and a first lower right corner coordinate of the third component, and a second upper left corner coordinate and a second lower right corner coordinate of the fourth component;

[0022] Determine a rectangular collision detection result according to a rectangular collision detection rule, the first upper left corner coordinate, the first lower right corner coordinate, the first upper left corner coordinate, and the second lower right corner coordinate;

[0023] Determine a rectangular coverage detection result according to a rectangular coverage detection rule, the first upper left corner coordinate, the first lower right corner coordinate, the first upper left corner coordinate, and the second lower right corner coordinate;

[0024] The rectangular collision detection result and the rectangular collision detection result are used as the detection result.

[0025] In a possible design, in another implementation of another aspect of the embodiment of the present application, the rectangular collision detection rule is that the left boundary of the first rectangle is smaller than the right boundary of the second rectangle, and the left boundary of the second rectangle is smaller than the right boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the horizontal direction, the left boundary is determined according to the horizontal axis value of the upper left corner coordinate of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the horizontal axis value of the lower right corner coordinate of the rectangle in the plane rectangular coordinate system;

[0026] and the upper boundary of the first rectangle is smaller than the lower boundary of the second rectangle, and the upper boundary of the second rectangle is smaller than the lower boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the vertical direction, the left boundary is determined according to the longitudinal axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the longitudinal axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system;

[0027] The rectangular coverage detection rule is that the left boundary of the first rectangle is less than or equal to the right boundary of the second rectangle, and the right boundary of the first rectangle is greater than or equal to the right boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the horizontal direction;

[0028] And if the upper boundary of the first rectangle is less than or equal to the upper boundary of the second rectangle, and the lower boundary of the first rectangle is greater than or equal to the lower boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the vertical direction.

[0029] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to sequentially obtain component types of the components;

[0030] When the component type is a picture type, obtaining a transparency value of a third component corresponding to the picture type;

[0031] When the transparency value is 0, the third component is filtered out;

[0032] When the transparency value is not 0, obtaining the color attribute value and the image area of ​​the third component;

[0033] When the color attribute value and the area of ​​the picture occupying the screen area meet a third preset condition, filtering out the third subcomponent;

[0034] When the color attribute value and the area of ​​the picture occupying the screen area do not satisfy a third preset condition, obtaining the coverage area of ​​the component is triggered.

[0035] In a possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to traverse all components on the target canvas and obtain parent components of the components in sequence;

[0036] Get the component type of the parent component;

[0037] When the component type of the parent component is a scrolling rectangle component, obtain the first coordinate area of ​​the parent component and the second coordinate area of ​​the component;

[0038] When the second coordinate area exceeds the first coordinate area, filtering out the component;

[0039] When the second coordinate area does not exceed the first coordinate area, the component is assigned to the at least one target component.

[0040] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to obtain the coordinates of the upper left corner and the lower right corner of the canvas;

[0041] Determine the width and height of the picture according to the upper left corner coordinate and the lower right corner coordinate;

[0042] The canvas area is calculated according to the width and the height.

[0043] In a possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to obtain a rendering order attribute sortingOrder value of each canvas according to the control tree information;

[0044] The first rendering priority of the plurality of canvases is determined according to the value of the rendering order attribute sortingOrder, wherein the larger the value of the rendering order attribute sortingOrder is, the higher the rendering priority of the canvas is.

[0045] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to obtain position information of the multiple components in the hierarchical list;

[0046] The second rendering priority of the multiple components is determined according to the position information, wherein the lower the position of a component in the hierarchical list is, the higher the rendering priority of the component is.

[0047] In a possible design, in another implementation of another aspect of the embodiment of the present application, the first preset condition is that the ratio of the canvas area to the screen area is greater than one quarter;

[0048] The second preset condition is that the ratio of the coverage area to the screen area is greater than one quarter.

[0049] Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;

[0050] Wherein, the memory is used to store programs;

[0051] The processor is used to execute the program in the memory, and the processor is used to execute the above-mentioned methods according to the instructions in the program code;

[0052] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0053] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0054] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0055] It can be seen from the above technical solution that the embodiment of the present application has the following advantages: according to the rendering priority of the canvas and in combination with traversing the canvas, the coverage ratio between the canvas area and the screen area is calculated to determine the top-level canvas (i.e., the first layer of canvas displayed on the current interface). In this way, all canvases displayed on the top layer can be accurately obtained, thereby filtering out redundant bottom-level canvases, and then filtering out components that affect redundancy, and ultimately improving the accuracy of component recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of an application system architecture of the method for identifying interface components in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of an embodiment of a method for identifying an interface component in an embodiment of the present application;

[0058] Figure 2a A schematic diagram of each canvas in the current interface in an embodiment of the present application;

[0059] Figure 2b A schematic diagram of the relationship between components in each canvas in an embodiment of the present application;

[0060] Figure 2c A schematic diagram of a flow chart for calculating the ratio of the canvas area to the screen area in an embodiment of the present application;

[0061] Figure 2d A schematic diagram of a flow chart of a method for identifying an interface component in an embodiment of the present application;

[0062] Figure 2e A schematic diagram of the effect between the identification method of the interface component in the embodiment of the present application and the existing local testing method;

[0063] Figure 3 This is another schematic diagram of an embodiment of the method for identifying an interface component in an embodiment of the present application;

[0064] Figure 3aA schematic diagram of a flow chart of the identification of interface components in an embodiment of the present application;

[0065] Figure 3b A schematic diagram of the effect between the interface component identification method in the embodiment of the present application and the existing interface component identification method;

[0066] Figure 4 This is another schematic diagram of an embodiment of the method for identifying an interface component in an embodiment of the present application;

[0067] Figure 4a A schematic diagram of rectangular collision detection in an embodiment of the present application;

[0068] Figure 4b This is a schematic diagram of an embodiment of rectangular coverage detection in an embodiment of the present application;

[0069] Figure 4c A schematic diagram of a flow chart of a method for identifying an interface component in an embodiment of the present application;

[0070] Figure 4d A schematic diagram of the effect between the method for identifying an interface component in the embodiment of the present application and the method for identifying an interface component in the prior art;

[0071] Figure 5 This is another schematic diagram of an embodiment of the method for identifying an interface component in an embodiment of the present application;

[0072] Figure 5a A schematic diagram of a flow chart of a method for identifying an interface component in an embodiment of the present application;

[0073] Figure 5b A schematic diagram of the effect between the method for identifying an interface component in the embodiment of the present application and the method for identifying an interface component in the prior art;

[0074] Figure 6 This is another schematic diagram of an embodiment of the method for identifying an interface component in an embodiment of the present application;

[0075] Figure 7 A schematic diagram of a flow chart of a method for identifying an interface component in an embodiment of the present application;

[0076] Figure 8 A schematic diagram of a flow chart of a method for identifying an interface component in an embodiment of the present application;

[0077] Fig. 9 A schematic diagram of an embodiment of an identification device for an interface component in an embodiment of the present application;

[0078] Fig.10 This is a schematic diagram of another embodiment of the identification device of the interface component in the embodiment of the present application;

[0079] Fig.11 This is a schematic diagram of another embodiment of the identification device of the interface component in the embodiment of the present application. DETAILED DESCRIPTION

[0080] The embodiments of the present application provide a method, apparatus, device and storage medium for identifying an interface component, which are used to improve the recognition accuracy of the interface component.

[0081] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0082] Under the tide of the Internet, games or other corresponding applications are increasingly inclined to global development. In this process, since the languages ​​used in the development of games or applications in different countries are generally the native languages ​​of the country, in order to meet the needs of the global development of games or applications, the languages ​​that appear in the game can generally be translated into other languages ​​so that the language texts of the corresponding languages ​​can be displayed in the game. In general, in game development testing, in order to ensure the translation quality of the language texts in the game, localization quality assurance (LQA) is required. The existing LQA needs to take screenshots of the language texts to be checked in the game or application, and the screenshots of the translated language texts in different languages ​​corresponding to the language texts to be checked are retrieved in sequence for comparison. At present, each component in the game interface or application interface is usually identified, and then the screenshots are reported. However, when performing component identification, it is possible to identify the unnecessary underlying components, making the component identification inaccurate. Therefore, how to quickly and accurately identify each component in the interface is an urgent problem to be solved.

[0083] In order to solve the above technical problems, the present application provides the following technical solutions: obtain the control tree information of the current interface, the control tree information is used to indicate the rendering priority of multiple canvases of the current interface, and the canvas is used to carry and render each component of the current interface; determine the first rendering priority of the multiple canvases according to the control tree information; traverse each canvas in order from high to low according to the first rendering priority, obtain the canvas area of ​​the canvas and calculate the ratio of the canvas area to the screen area, the screen area is the display area of ​​the display device that displays the current interface; when the ratio of the canvas area to the screen area meets the first preset condition, stop traversing the canvas and use the current canvas as the target canvas; obtain at least one target component on the target canvas. In this way, according to the rendering priority of the canvas and the coverage ratio between the canvas area and the screen area combined with the traversal of the canvas, the top-level canvas (i.e., the first layer of canvas displayed on the current interface) is determined. In this way, all displayed canvases at the top level can be accurately obtained, thereby filtering out redundant bottom-level canvases, and then filtering out redundant components, thereby improving the accuracy of component recognition.

[0084] The local testing methods of the optional embodiments of the present application can be implemented based on artificial intelligence technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0085] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0086] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0087] The present application also involves cloud technology, which refers to a hosting technology that unifies system resources such as hardware, software, and network in a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0088] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of Internet behavior, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing. The cloud technology involved in this application mainly refers to the transmission of game interfaces between terminal devices or servers through the "cloud", etc.

[0089] For ease of understanding, some nouns in this application are explained below.

[0090] Unity: is a real-time three-dimensional (3D) interactive content creation and operation platform. It can be applied to game development, art, architecture, automotive design, and film and television development. The platforms it supports include but are not limited to mobile phones, tablets, personal computers, game consoles, augmented reality and virtual reality devices.

[0091] Localization quality assurance (LQA): LQA is part of game testing or application testing, and it focuses on testing localization-related issues. The ultimate goal is to improve the experience of target language users from the perspectives of language and culture.

[0092] Canvas: Also known as canvas, it is used to carry and render components of user interface (UI) elements. Canvas can also be called a container for graphics, which is mainly used to draw graphics through javascript, that is, using scripts to draw graphics.

[0093] sortingOrder: It is a property used to control the rendering order in Unity. In Unity, the sortingOrder has the following characteristics: 1. In the same layer (also called the same canvas), if an image component and a particle component have the same sortingOrder, then the particle component will be located below the image component, that is, the particle component will be blocked by the image component. At this time, the image component and the particle component have nothing to do with the display state and display list order. 2. The sortingOrder of the parent component will affect all child components unless the child component sets the sortingOrder. That is, if the parent component has a sortingOrder, but the child component does not set the sortingOrder, then the sortingOrder of the child component is consistent with the sortingOrder of the parent component. Because the particle component has its own sortingOrder, the particle component is not affected by the sortingOrder of the parent component.

[0094] Mask: It is a UI component in Unity, used to create mask effects in UI.

[0095] Scroll Rect: is a UI component in Unity that is used to create a scrollable area in the UI.

[0096] Color: In Unity, Color is a structure used to represent color. It contains four floating-point components, representing red, green, blue and transparency.

[0097] The present application provides an interface component recognition method, device, equipment and storage medium, which can improve the recognition accuracy of the interface component. The following describes an exemplary application of the electronic device provided by the present application embodiment. The electronic device provided by the present application embodiment can be implemented as various types of user terminals or as a server.

[0098] By running the interface component identification scheme provided by the embodiment of the present application, the electronic device can ensure that the recognition accuracy of the interface components is improved, that is, the recognition accuracy of the electronic device itself for the interface components is improved, which is suitable for multiple application scenarios of local testing. For example, local testing of games, local testing of communication software, and local testing of shopping software can be performed. For example, the terminal device is a vehicle-mounted terminal, and the server performs component identification on the UI interface of each application applied to the vehicle-mounted terminal based on the above-mentioned interface component identification method, and then performs local testing. For another example, the terminal device is a smart phone, and the server performs component identification on the UI interface of each application applied to the smart phone based on the above-mentioned interface component identification method, and then performs local testing.

[0099] In one possible implementation, the interface component identification method provided in the embodiment of the present application can be implemented as an application or a part of an application and installed in a terminal, so that the terminal has the ability to schedule the application to perform component identification on each UI interface of other programs deployed on the terminal; the interface component identification method provided in the embodiment of the present application can be applied to a background server, so that the server schedules the application to perform component identification on each UI interface of other programs.

[0100] See also Figure 1 , Figure 1It is an optional architecture diagram under an application scenario of the identification method of the interface component provided in the embodiment of the present application. In order to implement the identification scheme supporting an interface component, the terminal device 100 is connected to the server 300 through the network 200, and the server 300 is connected to the database 400. The network 200 can be a wide area network or a local area network, or a combination of the two. The client for implementing the identification scheme of the interface component is deployed on the terminal device 100, wherein the client can be run on the terminal device 100 in the form of a browser, or can be run on the terminal device 100 in the form of an independent application (application, APP), etc. The specific presentation form of the client is not limited here. The server 300 involved in the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 100 can be a smart phone, a tablet computer, a laptop, a PDA, a personal computer, a smart TV, a smart watch, a vehicle-mounted device, a wearable device, an intelligent voice interaction device, a smart home appliance, an aircraft, etc., but is not limited thereto. The terminal device 100 and the server 300 can be directly or indirectly connected through the network 200 by wired or wireless communication, and this application does not limit this. The number of servers 300 and terminal devices 100 is also not limited. The solution provided in this application can be completed independently by the terminal device 100, or by the server 300, or by the terminal device 100 and the server 300 in cooperation, and this application does not make specific limitations on this. Among them, the database 400, in short, can be regarded as an electronic file cabinet-a place for storing electronic files, and users can add, query, update, delete, and other operations on the data in the file. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application. Database Management System (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, interception, security, and backup.Database management systems can be classified according to the database models they support, such as relational, Extensible Markup Language (XML); or according to the types of computers they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​at the same time. In the present application, database 400 can be used to store the current interface, the control tree information of the current interface, and the component information set. Of course, the storage location of the current interface, the control tree information of the current interface, and the component information set is not limited to the database. For example, it can also be stored in the terminal device 100, the blockchain, or the distributed file system of the server 300.

[0101] It can be understood that in the specific implementation of the present application, related data such as the current interface and the control tree information, component information set of the current interface, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0102] In combination with the above introduction, the following describes the identification method of the interface components in this application with the terminal device as the execution subject, please refer to Figure 2 In one embodiment of the present application, an embodiment of the method for identifying an interface component includes:

[0103] 201. Obtain control tree information of the current interface, where the control tree information is used to indicate rendering priorities of multiple canvases of the current interface, where the canvases are used to carry and render various components of the current interface.

[0104] When the program to be tested runs on a terminal device, different interfaces of the program at different running times will be displayed through the display device of the terminal device; then the terminal device can obtain the control tree information of the current interface.

[0105] It should be understood that for one interface, its corresponding different UI components will be carried by multiple canvases, such as Figure 2aIn the interface shown, the pop-up window displayed at the front end of the interface can be understood as a canvas, and the underlying background under the pop-up window can also be understood as another canvas. Based on the above structure, the control tree information can be used to indicate the rendering priority of different canvases in the current interface; or, the control tree information can also represent the sortingOrder value of each component in the current interface, the information of the canvas to which each component belongs, and the tree structure information between the parent component and the child component between each component.

[0106] 202. Determine a first rendering priority of the multiple canvases according to the control tree information.

[0107] In this embodiment, since the control tree information can be used to record the rendering priority of each canvas in the current interface or record the sortingOrder value of each UI component of the current interface, the terminal device can determine the rendering priority of each canvas of the current interface based on the control tree information and determine the rendering priority order.

[0108] In this embodiment, the higher the rendering priority, the closer the canvas is to the front of the current interface. Figure 2a As shown, the rendering priority of the canvas corresponding to the pop-up window is higher than the rendering priority of the canvas corresponding to the underlying background.

[0109] Optionally, when the control tree information records the sortingOrder values ​​of each UI component of the current interface, the rendering priority of each canvas in the current interface can be determined according to the characteristics of the sortingOrder value described above and the relationship between the parent component and the child component of each component. Figure 2b As shown, the pop-up window includes components 1, 2 and 3, and the underlying background image includes components 4, 5 and 6; wherein, component 1 is the parent component of components 2 and 3, and component 4 is the parent component of components 5 and 6, the sortingOrder value corresponding to component 1 is 4, and the sortingOrder value corresponding to component 4 is 2, then the rendering priority corresponding to component 1 is higher than that of component 4, and the corresponding rendering priority of components 2 and 3 is also higher than that of components 5 and 6. At the same time, since components 1, 2 and 3 are carried in the pop-up window, and components 4, 5 and 6 are carried in the underlying background image, the rendering priority of the pop-up window is higher than that of the underlying background image.

[0110] 203. Traverse each canvas in descending order of the first rendering priority, obtain the canvas area of ​​the canvas, and calculate the ratio of the canvas area to the screen area, where the screen area is the display area of ​​the display device displaying the current interface.

[0111] The terminal device sequentially traverses each canvas in the current interface in descending order of the first rendering priority, and the specific operations during the traversal are as follows:

[0112] During the traversal process, the terminal device obtains the corresponding canvas area for each canvas, and then calculates the ratio of the canvas area of ​​each canvas to the screen area.

[0113] Optionally, the specific process of calculating the ratio of the canvas area to the screen area can be found in Figure 2c As shown:

[0114] First, obtain the coordinates of the upper left corner and the lower right corner of the canvas; calculate the width and height of the canvas according to the coordinates of the upper left corner and the lower right corner of the canvas; then calculate the canvas area according to the width and the length; at the same time, obtain the width and height of the screen, and then calculate the screen area according to the width and the height; divide the canvas area by the screen area to get the ratio. It should be understood that although in Figure 2c The order of obtaining the canvas area and the screen area is limited in the text, but in actual applications, there is no time limit for obtaining the canvas area and the screen area. That is, the canvas area and the screen area can be obtained at the same time; or the screen area can be obtained first and then the canvas area.

[0115] The canvas area is equal to the width of the canvas multiplied by the height of the canvas; the screen area is equal to the width of the screen multiplied by the height of the screen.

[0116] 204. When the ratio of the canvas area to the screen area meets a first preset condition, stop traversing the canvas and use the current canvas as the target canvas.

[0117] After obtaining the ratio of each canvas area to the screen area, the terminal device compares the ratio with the first preset condition. When the first preset condition is met, the current canvas is used as the target canvas, and the traversal operation is terminated. In this way, the traversal is performed in descending order of rendering priority, and the canvas area is combined for screening, so that the top-level canvas can be better determined, thereby filtering to the bottom-level canvas, and then obtaining more accurate detection components, thereby improving the accuracy of local testing.

[0118] In this embodiment, the first preset condition can be set to a ratio of the canvas area to the screen area that is greater than one-fourth, wherein the one-fourth is set based on an empirical value in actual applications. It should be understood that the first preset condition can be adjusted according to different application scenarios as long as it can meet the recognition requirements of the interface component, and is not specifically limited here.

[0119] 205. Obtain at least one target component on the target canvas.

[0120] The terminal device identifies the coordinate area of ​​the component from the target canvas. In this embodiment, the terminal device can use a variety of identification methods, such as using an image recognition model for identification or identifying according to the coordinate values ​​recorded in the control tree information. The specific method is not limited here.

[0121] The following uses a game application scenario to illustrate the process of obtaining the at least one target component in this embodiment. Figure 2d As shown:

[0122] First, the terminal device obtains the control tree information of the current game interface; then, based on the sortingOrder in the control tree information, the canvas rendering sorting list of the current game interface is obtained, wherein the sorting order of the canvas rendering sorting list is sorted from high to low according to the rendering priority of the canvas; then, each canvas in the canvas rendering sorting list is traversed, and the canvas area and the screen area of ​​the canvas are obtained, and then the ratio of the canvas area to the screen area is calculated; then, it is determined whether the ratio exceeds a preset threshold; if so, the canvas traversal is stopped, and the canvas whose current ratio exceeds the preset threshold is confirmed to be the target canvas, also called the top canvas; if not, the canvas traversal continues.

[0123] By filtering the components of the current interface in the above manner, the effect achieved can be as follows: Figure 2e As shown, the Figure 2e (a) in the figure indicates the component identification result without filtering (in Figure 2e Each component is identified and displayed with a box in the figure). Figure 2e (b) in FIG. 1 is used to indicate the component identification result after filtering according to the rendering priority and the canvas area. Figure 2e It can be seen that the technical solution provided in this embodiment can effectively filter out the interference of underlying components on local testing, reduce the false alarm rate of local testing out of frame, and thus improve the accuracy of local testing.

[0124] After identifying at least one target component on the target canvas, the terminal device may perform a local test on the at least one target component.

[0125] In this embodiment, the at least one target component may be a text component, a picture component, etc.

[0126] In an exemplary application, if the local test is a local test of a game, the multiple canvases included in the current interface can be understood as multiple display pictures in the game interface, and the target component can be understood as the picture in the game that needs to be checked for local testing, that is, the corresponding picture in the display picture where the language text to be checked for translation quality is located (for example, in an interface, a pop-up window is the top display picture, and at this time, the text control in the pop-up window is the target component to be detected). The language text is used to represent the UI elements in the game interface (which can also be understood as text controls. For example, when a non-player character in the game interacts with the player in text, the displayed text information). Since the language text in the corresponding display picture in the game interface is the language text that needs to be checked for translation quality, the language type corresponding to the language text in the target component in the game interface can be recorded as the original language type, that is, the target component corresponding to the above game interface contains the original language type text. It should be understood that the local test on other application programs is similar to the local test in the game interface, that is, the target component corresponding to the above game interface can also be a picture of other application programs or software outside the game. For example, the components corresponding to each page in a certain shopping software.

[0127] It can be understood that the above local test check can be to check whether the content of the translated language text and the language text corresponding to the original language type match, and to check whether the translated language text and the UI are adapted, etc. Whether the translated language text and the user interface are adapted can include formal problems such as whether the translated language text appears out of the box, line breaks, or performance errors in the user interface, and can also include whether the translated language text is unified with the content to be expressed in the game interface.

[0128] Exemplarily, if the language text corresponding to a component in the game interface is the Chinese character "月饼" (moon cake), the UI icon corresponding to the language text is a round moon cake, and it is set to translate the language text from the original language type to the English language type, then it will be translated as "moon cake". Therefore, when performing the local test check, it is necessary to determine whether the language text corresponding to the translated English language type is "moon cake", and whether "moon cake" exceeds the original area in the component.

[0129] Based on the above solution, in order to better identify the target component, the present application also provides the following technical solution. For details, please refer to Figure 3 As shown, another embodiment of the method for identifying interface components in the embodiments of the present application includes:

[0130] Steps 301 to 304 are the same as steps 201 to 204, and will not be elaborated here specifically.

[0131] 305. Obtain a component information set on the target canvas, where the component information set is used to indicate rendering priorities of multiple components of the target canvas.

[0132] In this embodiment, the terminal device obtains a hierarchical list of the target canvas, where the hierarchical list is used to indicate the rendering priority order of each component in the target canvas, wherein the lower the position of a component in the hierarchical list is, the higher its corresponding rendering priority is.

[0133] 306. Determine a second rendering priority of the multiple components according to the component information set.

[0134] The terminal device uses the position information in the hierarchical list as the second rendering priority of the multiple components.

[0135] 307. Traverse each component in descending order of the second rendering priority, and obtain the coverage area of ​​the components in sequence.

[0136] The terminal device traverses the components in the target canvas in descending order of the second rendering priority, and the specific operations during the traversal are as follows:

[0137] During the traversal process, the terminal device obtains the corresponding coverage area of ​​each component, and then calculates the ratio of the coverage area of ​​each component to the screen area.

[0138] Optionally, the specific process of calculating the ratio of the coverage area to the screen area can be found in Figure 2 The details of step 203 are not repeated here.

[0139] 308. When there is a first component whose coverage area and screen area ratio satisfies a second preset condition, the second component and a component before the first component are taken as the at least one target component.

[0140] The terminal device compares the ratio of the coverage area of ​​each component to the screen area with the threshold in the second preset condition. If the ratio is greater than the threshold, the traversal operation is stopped, and the first component is determined to be the target component, and the components before the first component are also attributed to the target component. If the ratio is less than the threshold, the traversal operation is continued, and the first component is loaded and attributed to the target component.

[0141] The following uses a game application scenario to illustrate the process of obtaining the at least one target component in this embodiment. Figure 3a As shown:

[0142] The terminal device obtains a component information set corresponding to the target canvas; then determines a rendering priority sort list of the components based on the component information set; then traverses each component in the target canvas in order of rendering priority from high to low; during the traversal process, calculates the ratio of the coverage area of ​​the component to the screen area; determines whether the ratio of the coverage area of ​​the component to the screen area exceeds a threshold in a second preset condition, and if so, determines that the component and the components before the component are top-level components (i.e., a set of intermediate components); if not, continues to traverse the components.

[0143] By filtering the components of the current interface in the above manner, the effect achieved can be as follows: Figure 3b As shown, the Figure 3b (a) in the figure indicates the component identification result without filtering (in Figure 3b Each component is identified and displayed with a box in the figure). Figure 3b (b) in FIG. 1 is used to indicate the component recognition result after filtering according to the rendering priority and the canvas area and filtering the components in the same canvas. Figure 3b It can be seen that the technical solution provided in this embodiment can effectively filter out the interference of underlying components on local testing, reduce the false alarm rate of local testing out of frame, and thus improve the accuracy of local testing.

[0144] Based on the above solution, in order to better identify the target component, this application also provides the following technical solutions, please refer to Figure 4 As shown, another embodiment of the method for identifying an interface component in the embodiment of the present application includes:

[0145] Steps 401 to 407 are the same as steps 301 to 307, and are not described in detail here.

[0146] 408. When there is a second component whose coverage area and the screen area have a ratio that meets a second preset condition, the second component and components preceding the second component are taken as an intermediate component set.

[0147] The terminal device compares the ratio of the coverage area of ​​each component to the screen area with the threshold in the second preset condition. If the ratio is greater than the threshold, the traversal operation is stopped, and the first component is determined to be the intermediate component set (also called the top component set), and the components before the first component are also classified into the intermediate component set. If the ratio is less than the threshold, the traversal operation is continued, and the first component is loaded and classified into the intermediate component set.

[0148] 409. Perform rectangular collision detection and rectangular coverage detection on each component in the intermediate component set to filter out the blocked parent component, so as to obtain the at least one target component, where the parent component is the parent component in the intermediate component set.

[0149] The terminal device groups each component in the intermediate component set in pairs to obtain corresponding multiple groups of components, and then performs rectangular collision detection and rectangular overlap detection on the multiple groups of components to obtain detection results; when the detection result determines that two components in a group of components intersect or overlap, the parent component and child component relationship of the two components is determined, and if one of them is a parent component, the parent component is filtered out, the child component is retained, and the child component is attributed to the target component; the above operation is repeated until all components are detected and the at least one target component is obtained.

[0150] In this embodiment, the specific process of the rectangle collision detection and the rectangle coverage detection may be as follows:

[0151] 1. Rectangular collision detection

[0152] In the rectangular collision detection, it is necessary to first obtain the coordinates of the upper left corners and the lower right corners of the two rectangles, and then determine whether the two rectangles intersect according to the rectangular collision detection rules, wherein the rectangular collision detection rules are as follows: if the left boundary of the first rectangle is smaller than the right boundary of the second rectangle, and the left boundary of the second rectangle is smaller than the right boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the horizontal direction, and the left boundary is determined according to the horizontal axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the horizontal axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system; and if the upper boundary of the first rectangle is smaller than the lower boundary of the second rectangle, and the upper boundary of the second rectangle is smaller than the lower boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the vertical direction, and the left boundary is determined according to the vertical axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the vertical axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system. For example, Figure 4a As shown, the coordinates of the upper left corner of the first rectangle are (10, 22), and the coordinates of the lower right corner are (25, 10). The coordinates of the upper left corner of the second rectangle are (15, 20), and the coordinates of the lower right corner are (30, 15). Then the distance between the left boundary of the first rectangle and the origin of the plane rectangular coordinate system is 10, the distance between the right boundary and the origin of the plane rectangular coordinate system is 25, the distance between the upper boundary and the origin of the plane rectangular coordinate system is 22, and the distance between the lower boundary and the origin of the plane rectangular coordinate system is 10. The distance between the left boundary of the second rectangle and the origin of the plane rectangular coordinate system is 15, the distance between the right boundary and the origin of the plane rectangular coordinate system is 30, the distance between the upper boundary and the origin of the plane rectangular coordinate system is 20, and the distance between the lower boundary and the origin of the plane rectangular coordinate system is 15. Then, according to the above collision detection rules, it can be determined that the first rectangle intersects with the second rectangle in the horizontal direction.

[0153] 2. Rectangular Coverage Detection

[0154] In the rectangle collision detection, it is necessary to first obtain the coordinates of the upper left corners and the lower right corners of the two rectangles, and then determine whether the two rectangles intersect according to the rectangle coverage detection rule, where the rectangle collision detection rule is as follows: the rectangle coverage detection rule is that if the left boundary of the first rectangle is less than or equal to the right boundary of the second rectangle, and the right boundary of the first rectangle is greater than or equal to the right boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the horizontal direction; and if the upper boundary of the first rectangle is less than or equal to the upper boundary of the second rectangle, and the lower boundary of the first rectangle is greater than or equal to the lower boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the vertical direction. For example, Figure 4b As shown, the coordinates of the upper left corner of the first rectangle are (10, 22), and the coordinates of the lower right corner are (25, 10). The coordinates of the upper left corner of the second rectangle are (15, 20), and the coordinates of the lower right corner are (20, 15). Then the distance between the left boundary of the first rectangle and the origin of the plane rectangular coordinate system is 10, the distance between the right boundary and the origin of the plane rectangular coordinate system is 25, the distance between the upper boundary and the origin of the plane rectangular coordinate system is 22, and the distance between the lower boundary and the origin of the plane rectangular coordinate system is 10. The distance between the left boundary of the second rectangle and the origin of the plane rectangular coordinate system is 15, the distance between the right boundary and the origin of the plane rectangular coordinate system is 20, the distance between the upper boundary and the origin of the plane rectangular coordinate system is 20, and the distance between the lower boundary and the origin of the plane rectangular coordinate system is 15. Then according to the above collision detection rules, it can be determined that the first rectangle covers the second rectangle.

[0155] The following uses a game application scenario to illustrate the process of obtaining the at least one target component in this embodiment. Figure 4c As shown:

[0156] The terminal device obtains a component information set corresponding to the target canvas; then determines a rendering priority sort list of the components based on the component information set; then traverses each component in the target canvas in order of rendering priority from high to low; calculates the ratio of the coverage area of ​​the component to the screen area during the traversal process; determines whether the ratio of the coverage area of ​​the component to the screen area exceeds the threshold in the second preset condition, and if so, determines that the component and the components before the component are top-level components (i.e., a set of intermediate components); if not, continues to traverse the components; performs rectangular collision detection and rectangular coverage detection on the set of intermediate components to obtain detection results; determines whether there are components in the set of intermediate components that are blocked based on the detection results, and if so, filters out the blocked components, and if not, loads the components to belong to the at least one target component.

[0157] By filtering the components of the current interface in the above manner, the effect achieved can be as follows: Figure 4d As shown, the Figure 4d (a) in the figure indicates the component identification result without filtering (in Figure 4d Each component is identified and displayed with a box in the figure). Figure 4d (b) in FIG. 1 is used to indicate the component recognition result after filtering according to the rendering priority and the canvas area, filtering the same layer canvas, and filtering the rectangular collision and rectangular coverage. Figure 4d It can be seen that the technical solution provided in this embodiment can effectively filter out the interference of underlying components on local testing, reduce the false alarm rate of local testing out of frame, and thus improve the accuracy of local testing.

[0158] exist Figure 3 and Figure 4 The solution shown does not mention filtering of transparent components. When adding transparent components in the local test, it will also affect the out-of-frame false alarm rate of the local test. Therefore, it is also necessary to filter the transparent components in the target canvas. The specific process can be as follows: Figure 5 As shown:

[0159] First, the terminal device traverses the components in the target canvas, and obtains the corresponding component type for each component; determines whether the component type is a picture type, and if so, obtains the transparency value of the component. If the transparency value is 0, it means that the component is a transparent component, and the component can be directly filtered out at this time; if the transparency value is not 0, calculate the color attribute value of the component (that is, calculate the color of each pixel in the component) and the picture area of ​​the component; determines whether it is a semi-transparent picture, that is, compares the color attribute value and the picture area with the expected conditions to obtain a comparison result, if the comparison result indicates that there is a transparent part in the component (for example, the color attribute value is less than 0.2), and the ratio of the picture area of ​​the component to the screen area is greater than a preset threshold (for example, the ratio is greater than 15%), then filter out the component; if the comparison result indicates that there is no transparent part in the component or the ratio of the picture area of ​​the component to the screen area is less than the preset threshold, load the component.

[0160] It is understandable that the Figure 5 The process shown can be added before traversing the area of ​​each component in the target canvas, or can be added in other stages that do not affect the filtering effect, which is not limited here. Figure 5The process shown is added before calculating the coverage area of ​​each component in the target canvas. When the comparison result indicates that there is no transparent part in the component or the ratio of the image area of ​​the component to the screen area is less than a preset threshold, it is calculated whether the ratio of the coverage area of ​​the component to the screen area meets the second preset condition (for example, whether the ratio is greater than one quarter).

[0161] The following is Figure 5 The process shown is added before the area traversal of each component in the target canvas, taking game testing as an example, Figure 5a As shown, the acquisition process of the at least one target component is described:

[0162] The terminal device obtains a component information set corresponding to the target canvas; then determines a rendering priority sort list of the components based on the component information set; then traverses each component in the target canvas in descending order of rendering priority; during the traversal process, first determines whether the component type of the component is a picture type; if so, determines whether the transparency of the component is 0; if not, calculates the ratio of the coverage area of ​​the component to the screen area; when the transparency of the component is 0, filters out the component; when the transparency of the component is not 0, calculates the color attribute value of the component and the ratio of the area of ​​the group The image area of ​​the component; when the color attribute value is less than the first threshold value and the ratio of the image area to the screen area exceeds the second threshold value, it is determined that the component has a transparent part, and the component is filtered out at this time; if it is determined that the component does not have a transparent part according to the color attribute value and the image area, the ratio of the coverage area of ​​the component to the screen area is calculated; it is determined whether the ratio of the coverage area of ​​the component to the screen area exceeds a third threshold value, if so, it is determined that the component and the components before the component are top-level components (that is, the at least one target component), if not, the components are continued to be traversed.

[0163] By filtering the components of the current interface in the above manner, the effect achieved can be as follows: Figure 5b As shown, the Figure 5b (a) in the figure indicates the component identification result without filtering (in Figure 5b Each component is identified and displayed with a box in the figure). Figure 5b (b) in the figure indicates the component recognition result after filtering according to the rendering priority and the canvas area, filtering the same layer canvas, filtering the rectangular collision, filtering the rectangular coverage, and filtering the transparent components. Figure 4d It can be seen that the technical solution provided in this embodiment can effectively filter out the interference of underlying components on local testing, reduce the false alarm rate of local testing out of frame, and thus improve the accuracy of local testing.

[0164] Through the above Figures 2 to 5Any of the above solutions can filter static components. When there are scrollable components in the target canvas, Figures 2 to 5 The solution in the application cannot handle it, so the application also provides a solution for filtering scrollable components. The specific process can be as follows: Figure 6 As shown:

[0165] First, the terminal device traverses all components on the target canvas, and determines the corresponding parent component for each component; then obtains the component type of the parent component; if the component type of the parent component is a scrollable rectangular component, obtains the first coordinate area of ​​the parent component and the second coordinate area of ​​the component; then determines whether the second coordinate area exceeds the first coordinate area; if so, filters out the component; if not, loads the component and uses the component as a candidate component for the at least one target component.

[0166] It should be understood that when determining whether the coordinate area of ​​the child component exceeds the coordinate area of ​​the parent component, the rectangular collision detection and rectangular overlap detection described above can also be used, and the details will not be repeated here.

[0167] It should be understood that in the operation process of obtaining the at least one target component, the Figure 6 The solution shown does not limit the stage of the operation process in which it is added, as long as the scrollable component can be filtered. Figure 6 The solution shown is the last step in the operation process of obtaining the target component, and the component that does not exceed the coordinate area of ​​the parent component is directly attributed to the target component. In another exemplary solution, if the Figure 6 The solution shown is an intermediate step in the operation process of obtaining the target component, and the component that does not exceed the coordinate area of ​​the parent component can be classified as a candidate component of the target component.

[0168] The effect of the filtering solution of the scrollable component can be shown as follows Figure 7 As shown, when the scrollable component is not filtered, the target component identified by the terminal device can be as follows Figure 7 As shown in (a) Figure 7 Each component is identified and displayed as a box in the parent component), that is, components that exceed the display area of ​​the parent component will also be reported, and then during local testing, an over-box warning may appear. After filtering the scrollable component, the target component identified by the terminal device can be as follows Figure 7 As shown in (b), components that exceed the display area of ​​the parent component are directly filtered out and no local testing is performed. This can greatly reduce the false alarm rate of out-of-frame.

[0169] Combined with the above Figures 2 to 6The technical solution described in the following takes the game interface as an example. Figure 8 As shown, the identification method of the interface component is described:

[0170] The acquisition process is mainly divided into three parts. One part is used to filter different canvases in the game interface, that is, the process of obtaining the target canvas; another part is used to filter transparent components and overlay components of different components in the same canvas in the game interface; another part is used to filter scrollable components in the same canvas in the game interface.

[0171] In the first part, the terminal device obtains the control tree information of the current game interface; then obtains the canvas rendering sort list of the current game interface based on the control tree information, wherein the sorting order of the canvas rendering sort list is sorted from high to low according to the rendering priority of the canvas; then traverses each canvas in the canvas rendering sort list, and obtains the canvas area and screen area of ​​the canvas, and then calculates the ratio of the canvas area to the screen area; then determines whether the ratio exceeds a preset threshold; if so, stops canvas traversal, and confirms that the canvas whose current ratio exceeds the preset threshold is the target canvas, also called the top canvas; if not, continues to traverse the canvas.

[0172] In the second part, the terminal device obtains a component information set corresponding to the target canvas; then determines a rendering priority sort list of the components based on the component information set; and then traverses each component in the target canvas in descending order of rendering priority; during the traversal process, first determine whether the component type of the component is a picture type, and if so, determine whether the transparency of the component is 0, and if not, calculate the ratio of the coverage area of ​​the component to the screen area; when the transparency of the component is 0, filter out the component, and when the transparency of the component is not 0, calculate the color attribute value of the component and the picture area of ​​the component; when the color attribute value is less than the first threshold value, and the ratio of the picture area to the screen area exceeds the second threshold value , determine that the component has a transparent part, and filter out the component at this time; if it is determined that the component does not have a transparent part according to the color attribute value and the image area, calculate the ratio of the coverage area of ​​the component to the screen area; determine whether the ratio of the coverage area of ​​the component to the screen area exceeds a third threshold, if so, determine that the component and the components before the component are top-level components (i.e., a set of intermediate components), if not, continue to traverse the components; perform rectangular collision detection and rectangular coverage detection on the set of intermediate components to obtain detection results; determine whether there is a component in the set of intermediate components that is blocked according to the detection result, if so, filter out the blocked component, if not, load the component to belong to the at least one target component.

[0173] In the third part, the terminal device traverses all components on the target canvas and obtains the parent component of the component; then obtains the component type of the parent component; when the component type of the parent component is a scrolling rectangular component, obtains the first coordinate area of ​​the parent component and the second coordinate area of ​​the component, and loads the component when the component type of the parent component is a non-scrolling rectangular component; when the second coordinate area exceeds the first coordinate area, filters out the component, and when the second coordinate area does not exceed the first coordinate area, attributes the component to the at least one target component.

[0174] The following is a detailed description of the identification device of the interface component in this application. Fig. 9 , Fig. 9 This is a schematic diagram of an embodiment of an interface component identification device in an embodiment of the present application. The interface component identification device 20 includes:

[0175] The acquisition module 201 is used to acquire the control tree information of the current interface, where the control tree information is used to indicate the rendering priority of multiple canvases of the current interface, where the canvas is used to carry and render various components of the current interface;

[0176] The processing module 202 is used to determine the first rendering priority of the multiple canvases according to the control tree information; traverse each canvas in order from high to low according to the first rendering priority, obtain the canvas area of ​​the canvas and calculate the ratio of the canvas area to the screen area, where the screen area is the display area of ​​the display device that displays the current interface; when the ratio of the canvas area to the screen area meets a first preset condition, stop traversing the canvas and use the current canvas as the target canvas; and obtain at least one target component on the target canvas.

[0177] In an embodiment of the present application, a device for identifying interface components is provided. Using the above device, the coverage ratio between the canvas area and the screen area is calculated based on the rendering priority of the canvas and combined with the traversal of the canvas, so as to determine the top-level canvas (i.e., the first layer of canvas displayed on the current interface). In this way, all canvases displayed on the top layer can be accurately obtained, thereby filtering out redundant bottom-level canvases, and then filtering out redundant components, and finally improving the recognition accuracy of interface components.

[0178] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0179] The processing module 202 is used to obtain a component information set on the target canvas, where the component information set is used to indicate the rendering priority of multiple components of the target canvas;

[0180] Determine a second rendering priority of the multiple components according to the component information set;

[0181] Traverse each component in descending order of the second rendering priority to obtain the coverage area of ​​the component;

[0182] When there is a first component whose coverage area and the screen area have a ratio that meets the second preset condition, the first component and the component before the first component are taken as the at least one target component.

[0183] In an embodiment of the present application, a device for identifying interface components is provided. Using the above device, different components on the top canvas are further screened by rendering priority, that is, the components are traversed to calculate the coverage ratio between the coverage area of ​​the component and the screen area, so as to determine the top components and valid components. In this way, valid components can be accurately obtained, thereby filtering out redundant components, and ultimately improving the recognition accuracy of interface components.

[0184] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the device 20 for identifying an interface component provided by the embodiment of the present application, the processing module 202 is used to obtain a component information set on the target canvas, and the component information set is used to indicate the rendering priority of multiple components of the target canvas;

[0185] Determine a second rendering priority of the multiple components according to the component information set;

[0186] Traverse each component in descending order of the second rendering priority, and obtain the coverage area of ​​the components in sequence;

[0187] When there is a second component whose coverage area and the screen area have a ratio that satisfies a second preset condition, the second component and the components before the second component are taken as an intermediate component set;

[0188] Rectangular collision detection and rectangular coverage detection are performed on each component in the intermediate component set to filter out the blocked parent component, so as to obtain the at least one target component, which is the parent component in the intermediate component set.

[0189] In an embodiment of the present application, a device for identifying interface components is provided. Using the above device, different components on the top canvas are further screened by rendering priority, that is, the components are traversed to calculate the coverage ratio between the coverage area of ​​the component and the screen area, so as to determine the top components and valid components. At the same time, rectangular collision detection and rectangular coverage detection are combined, so that valid components can be accurately obtained, thereby filtering out redundant components, and finally improving the recognition accuracy of interface components.

[0190] Optionally, in the above Fig. 9On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0191] The processing module 202 is used to perform rectangular collision detection and rectangular coverage detection on a first group of components in the intermediate component set to obtain detection results, wherein the first group of components includes a third component and a fourth component, and the first group of components is obtained by grouping in pairs;

[0192] When the detection result determines that there is a rectangular collision or rectangular overlap between the third component and the fourth component, determine the parent component and child component relationship between the third component and the fourth component, filter out the parent components of the third component and the fourth component, and use the child components of the third component and the fourth component as the target component;

[0193] Repeat the above operation until the set of intermediate components is traversed and the at least one target component is obtained.

[0194] In an embodiment of the present application, a device for identifying interface components is provided. Using the above device, for two components that have a rectangular collision or rectangular overlap, the parent component is filtered out, so that the displayed component can be guaranteed to be the top-level component, so that the valid component can be accurately obtained, and then the redundant components are filtered out, and finally the recognition accuracy of the interface component is improved.

[0195] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0196] The processing module 202 is used to obtain the first upper left corner coordinate and the first lower right corner coordinate of the third component, and the second upper left corner coordinate and the second lower right corner coordinate of the fourth component;

[0197] Determine a rectangular collision detection result according to a rectangular collision detection rule, the first upper left corner coordinate, the first lower right corner coordinate, the first upper left corner coordinate, and the second lower right corner coordinate;

[0198] Determine a rectangular coverage detection result according to a rectangular coverage detection rule, the first upper left corner coordinate, the first lower right corner coordinate, the first upper left corner coordinate, and the second lower right corner coordinate;

[0199] The rectangular collision detection result and the rectangular collision detection result are used as the detection result.

[0200] In an embodiment of the present application, a device for identifying interface components is provided. By using the device, the positional relationship between two components is determined according to the coordinate positions of the components, so that rectangular collision and rectangular coverage detection can be conveniently and effectively implemented.

[0201] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the identification device 20 of the interface component provided by the embodiment of the present application, the rectangular collision detection rule is that the left boundary of the first rectangle is smaller than the right boundary of the second rectangle, and the left boundary of the second rectangle is smaller than the right boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the horizontal direction, and the left boundary is determined according to the horizontal axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the horizontal axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system;

[0202] and the upper boundary of the first rectangle is smaller than the lower boundary of the second rectangle, and the upper boundary of the second rectangle is smaller than the lower boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the vertical direction, the left boundary is determined according to the longitudinal axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the longitudinal axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system;

[0203] The rectangular coverage detection rule is that the left boundary of the first rectangle is less than or equal to the right boundary of the second rectangle, and the right boundary of the first rectangle is greater than or equal to the right boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the horizontal direction;

[0204] And if the upper boundary of the first rectangle is less than or equal to the upper boundary of the second rectangle, and the lower boundary of the first rectangle is greater than or equal to the lower boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the vertical direction.

[0205] In an embodiment of the present application, a device for identifying interface components is provided. By using the device, the positional relationship between two components is determined according to the coordinate values ​​in a plane coordinate system, so that rectangular collision and rectangular coverage detection can be conveniently and effectively implemented.

[0206] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the device 20 for identifying an interface component provided by the embodiment of the present application, the processing module 202 is used to sequentially obtain the component types of the components;

[0207] When the component type is a picture type, obtaining a transparency value of a third component corresponding to the picture type;

[0208] When the transparency value is 0, the third component is filtered out;

[0209] When the transparency value is not 0, obtaining the color attribute value and the image area of ​​the third component;

[0210] When the color attribute value and the area of ​​the picture occupying the screen area meet a third preset condition, filtering out the third subcomponent;

[0211] When the color attribute value and the area of ​​the picture occupying the screen area do not satisfy a third preset condition, obtaining the coverage area of ​​the component is triggered.

[0212] In an embodiment of the present application, a device for identifying interface components is provided. By using the above device, transparent components can be filtered by judging the transparency and color attribute values ​​of the image, thereby reducing the interference of transparent components on the identification of interface components, thereby improving the recognition accuracy of interface components.

[0213] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0214] The processing module 202 is used to traverse all components on the target canvas and obtain the parent components of the components in sequence;

[0215] Get the component type of the parent component;

[0216] When the component type of the parent component is a scrolling rectangle component, obtain the first coordinate area of ​​the parent component and the second coordinate area of ​​the component;

[0217] When the second coordinate area exceeds the first coordinate area, filtering out the component;

[0218] When the second coordinate area does not exceed the first coordinate area, the component is assigned to the at least one target component.

[0219] In an embodiment of the present application, a device for identifying an interface component is provided. The above device is used to calculate the coordinate area of ​​the scrolling component and the coordinate area of ​​the subcomponent of the scrolling component, thereby calculating whether the subcomponent exceeds the display area of ​​the scrolling component. If it exceeds, the subcomponent is filtered out, which can reduce the recognition of components that are not fully displayed, thereby improving the recognition accuracy of the interface component.

[0220] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0221] The processing module 202 is used to obtain the coordinates of the upper left corner and the lower right corner of the canvas;

[0222] Determine the width and height of the picture according to the upper left corner coordinate and the lower right corner coordinate;

[0223] The canvas area is calculated according to the width and the height.

[0224] In the embodiment of the present application, a device for identifying an interface component is provided. By using the device, a specific solution for calculating the canvas area is provided, which can increase the feasibility of the solution.

[0225] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0226] The processing module 202 is used to obtain the rendering order attribute sortingOrder value of each canvas according to the control tree information;

[0227] The first rendering priority of the plurality of canvases is determined according to the value of the rendering order attribute sortingOrder, wherein the larger the value of the rendering order attribute sortingOrder is, the higher the rendering priority of the canvas is.

[0228] In an embodiment of the present application, a device for identifying an interface component is provided. By using the above device, the sortingOrder value of each canvas is obtained through the control tree information. This attribute value is a fixed value, so the rendering priority of each canvas can be accurately obtained according to the above sortingOrder value, thereby increasing the feasibility of the solution.

[0229] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0230] The processing module 202 is used to obtain the position information of the multiple components in the hierarchical structure list;

[0231] The second rendering priority of the multiple components is determined according to the position information, wherein the lower the position of a component in the hierarchical list is, the higher the rendering priority of the component is.

[0232] In an embodiment of the present application, a device for identifying interface components is provided. By using the above device, the location information of each component is obtained through the hierarchy list of the canvas. Since the location information indicates the rendering priority of each component, the rendering priority of each component can be accurately obtained according to the hierarchy list, thereby increasing the feasibility of the solution.

[0233] Optionally, in the above Fig. 9 On the basis of the corresponding embodiment, in another embodiment of the interface component identification device 20 provided in the embodiment of the present application,

[0234] The first preset condition is that the ratio of the canvas area to the screen area is greater than one quarter;

[0235] The second preset condition is that the ratio of the coverage area to the screen area is greater than one quarter.

[0236] In an embodiment of the present application, a device for identifying an interface component is provided. By using the device, the coverage ratio of the canvas area to the screen area and the coverage ratio of the component coverage area to the screen area are set according to actual experience values, thereby increasing the feasibility of the solution.

[0237] The identification device of the interface component provided in this application can be used in the server, please refer to Fig.10 , Fig.10 : is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0238] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0239] The steps performed by the server in the above embodiment can be based on the Fig.10 The server structure shown.

[0240] The identification device of the interface component provided in this application can be used in terminal equipment, please refer to Fig.11 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. In the embodiments of the present application, the terminal device is a smart phone as an example for explanation:

[0241] Fig.11The block diagram shows a partial structure of a smart phone related to the terminal device provided in the embodiment of the present application. Fig.11 The smart phone includes: a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will appreciate that Fig.11 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0242] Combine the following Fig.11 A detailed introduction to the various components of a smartphone:

[0243] The RF circuit 410 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 480 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 410 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 410 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0244] The memory 420 can be used to store software programs and modules. The processor 480 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0245] The input unit 430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smart phone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 431 or near the touch panel 431 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 480, and can receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0246] The display unit 440 can be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 440 may include a display panel 441. Optionally, the display panel 441 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 431 may cover the display panel 441. When the touch panel 431 detects a touch operation on or near it, it is transmitted to the processor 480 to determine the type of touch event. Subsequently, the processor 480 provides a corresponding visual output on the display panel 441 according to the type of touch event. Although in Fig.11 In the embodiment, the touch panel 431 and the display panel 441 are used as two independent components to implement the input and output functions of the smartphone, but in some embodiments, the touch panel 431 and the display panel 441 can be integrated to implement the input and output functions of the smartphone.

[0247] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 441 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in smartphones, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0248] The audio circuit 460, the speaker 461, and the microphone 462 can provide an audio interface between the user and the smartphone. The audio circuit 460 can transmit the received audio data to the speaker 461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 460 and converted into audio data, and then the audio data is output to the processor 480 for processing, and then sent to another smartphone through the RF circuit 410, or the audio data is output to the memory 420 for further processing.

[0249] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse web pages, and access streaming media through WiFi module 470, which provides users with wireless broadband Internet access. Fig.11 A WiFi module 470 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.

[0250] Processor 480 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in memory 420, and calling data stored in memory 420, thereby monitoring the smartphone as a whole. Optionally, processor 480 may include one or more processing units; optionally, processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into processor 480.

[0251] The smart phone also includes a power source 490 (such as a battery) for supplying power to various components. Optionally, the power source can be logically connected to the processor 480 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0252] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0253] The steps performed by the terminal device in the above embodiment can be based on the Fig.11 The terminal device structure shown.

[0254] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0255] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0257] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0260] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0261] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying an interface component, characterized in that: include: Obtaining control tree information of the current interface, where the control tree information is used to indicate rendering priorities of multiple canvases of the current interface, where the canvases are used to carry and render various components of the current interface; Determine a first rendering priority of the plurality of canvases according to the control tree information; Traversing each canvas in descending order of the first rendering priority, obtaining a canvas area of ​​the canvas and calculating a ratio of the canvas area to a screen area, where the screen area is a display area of ​​a display device displaying the current interface; When the ratio of the canvas area to the screen area meets a first preset condition, stop traversing the canvas and use the current canvas as the target canvas; Get at least one target component on the target canvas.

2. The method according to claim 1, characterized in that The acquiring at least one target component on the target canvas comprises: Acquire a component information set on the target canvas, where the component information set is used to indicate rendering priorities of multiple components of the target canvas; Determining a second rendering priority of the plurality of components according to the component information set; Traverse each component in descending order of the second rendering priority to obtain the coverage area of ​​the component; When there is a first component whose coverage area and the screen area have a ratio that meets a second preset condition, the first component and a component preceding the first component are taken as the at least one target component.

3. The method according to claim 1, characterized in that: The acquiring at least one target component on the target canvas comprises: Acquire a component information set on the target canvas, where the component information set is used to indicate rendering priorities of multiple components of the target canvas; Determining a second rendering priority of the plurality of components according to the component information set; Traversing each component in descending order of the second rendering priority, and obtaining the coverage area of ​​the components in sequence; When there is a second component whose coverage area and the screen area have a ratio that meets a second preset condition, taking the second component and the components before the second component as an intermediate component set; Rectangular collision detection and rectangular coverage detection are performed on each component in the intermediate component set to filter out the blocked parent component, so as to obtain the at least one target component, where the parent component is the parent component in the intermediate component set.

4. The method according to claim 3, characterized in that The performing rectangular collision detection and rectangular coverage detection on each component in the intermediate component set to filter out the blocked parent component to obtain the at least one target component includes: Performing rectangular collision detection and rectangular coverage detection on a first group of components in the intermediate component set to obtain detection results, wherein the first group of components includes a third component and a fourth component, and the first components are obtained by grouping in pairs; When the detection result determines that there is a rectangular collision or rectangular overlap between the third component and the fourth component, determine the parent component and child component relationship between the third component and the fourth component, filter out the parent components in the third component and the fourth component, and use the child components in the third component and the fourth component as the target components; The above operation is repeated until the intermediate component set is traversed and the at least one target component is obtained.

5. The method according to claim 4, characterized in that The performing of rectangular collision detection and rectangular coverage detection on the first group of components in the intermediate component set to obtain detection results includes: Obtain the first upper left corner coordinate and the first lower right corner coordinate of the third component, and the second upper left corner coordinate and the second lower right corner coordinate of the fourth component; Determine a rectangular collision detection result according to a rectangular collision detection rule, the first upper left corner coordinate, the first lower right corner coordinate, the first upper left corner coordinate, and the second lower right corner coordinate; Determine a rectangular coverage detection result according to a rectangular coverage detection rule, the first upper left corner coordinate, the first lower right corner coordinate, the first upper left corner coordinate, and the second lower right corner coordinate; The rectangular collision detection result and the rectangular collision detection result are used as the detection result.

6. The method according to claim 5, characterized in that The rectangular collision detection rule is that the left boundary of the first rectangle is smaller than the right boundary of the second rectangle, and the left boundary of the second rectangle is smaller than the right boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the horizontal direction, the left boundary is determined according to the horizontal axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the horizontal axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system; and the upper boundary of the first rectangle is smaller than the lower boundary of the second rectangle, and the upper boundary of the second rectangle is smaller than the lower boundary of the first rectangle, then the first rectangle overlaps with the second rectangle in the vertical direction, the left boundary is determined according to the longitudinal axis value of the coordinates of the upper left corner of the rectangle in the plane rectangular coordinate system, and the right boundary is determined according to the longitudinal axis value of the coordinates of the lower right corner of the rectangle in the plane rectangular coordinate system; The rectangle coverage detection rule is that the left boundary of the first rectangle is less than or equal to the right boundary of the second rectangle, and the right boundary of the first rectangle is greater than or equal to the right boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the horizontal direction; And if the upper boundary of the first rectangle is less than or equal to the upper boundary of the second rectangle, and the lower boundary of the first rectangle is greater than or equal to the lower boundary of the second rectangle, then the first rectangle completely covers the second rectangle in the vertical direction.

7. The method according to any one of claims 2 to 6, characterized in that Before sequentially acquiring the coverage areas of the components, the method further includes: Get the component type of the component in turn; When the component type is a picture type, obtaining a transparency value of a third component corresponding to the picture type; When the transparency value is 0, filtering out the third component; When the transparency value is not 0, obtaining the color attribute value and the image area of ​​the third component; When the color attribute value and the area of ​​the picture occupying the screen area meet a third preset condition, filtering out the third subcomponent; When the color attribute value and the area of ​​the picture occupying the screen area do not satisfy a third preset condition, the acquisition of the coverage area of ​​the component is triggered.

8. The method according to any one of claims 2 to 6, characterized in that The method further comprises: Traverse all components on the target canvas and obtain the parent components of the components in turn; Get the component type of the parent component; When the component type of the parent component is a scrolling rectangle component, obtaining a first coordinate area of ​​the parent component and a second coordinate area of ​​the component; When the second coordinate area exceeds the first coordinate area, filtering out the component; When the second coordinate area does not exceed the first coordinate area, the component is attributed to the at least one target component.

9. The method according to any one of claims 1 to 6, characterized in that Getting the canvas area includes: Get the coordinates of the upper left corner and lower right corner of the canvas; Determine the width and height of the picture according to the upper left corner coordinates and the lower right corner coordinates; The canvas area is calculated according to the width and the height.

10. The method according to any one of claims 1 to 6, characterized in that Determining the first rendering priority of the plurality of canvases according to the control tree information comprises: Obtain the sortingOrder value of the rendering order attribute of each canvas according to the control tree information; A first rendering priority of the plurality of canvases is determined according to the value of the rendering order attribute sortingOrder, wherein the larger the value of the rendering order attribute sortingOrder is, the higher the rendering priority of the canvas is.

11. The method according to any one of claims 2 to 6, characterized in that The component information set is a hierarchy list of canvases, and determining the second rendering priority of the plurality of components according to the component information set includes: Obtaining position information of the plurality of components in the hierarchical list; The second rendering priority of the plurality of components is determined according to the position information, wherein the lower the position of a component in the hierarchy list is, the higher the rendering priority of the component is.

12. The method according to any one of claims 2 to 6, characterized in that The first preset condition is that the ratio of the canvas area to the screen area is greater than one quarter; The second preset condition is that the ratio of the coverage area to the screen area is greater than one quarter.

13. An identification device for an interface component, characterized in that: include: An acquisition module, used to acquire control tree information of the current interface, wherein the control tree information is used to indicate the rendering priority of multiple canvases of the current interface, and the canvas is used to carry and render various components of the current interface; a processing module, configured to determine a first rendering priority of the plurality of canvases according to the control tree information; traverse each canvas in descending order of the first rendering priority, obtain a canvas area of ​​the canvas and calculate a ratio of the canvas area to a screen area, wherein the screen area is a display area of ​​a display device that displays the current interface; and when the ratio of the canvas area to the screen area satisfies a first preset condition, stop traversing the canvases and use the current canvas as a target canvas; Get at least one target component on the target canvas.

14. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the method according to any one of claims 1 to 12 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

15. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 12.

Images