Method and system for extracting elements based on terminal screenshot picture

Abstract

The invention provides a method and a system for extracting elements based on a terminal screenshot picture. The method comprises the following steps: acquiring a screenshot containing a complete target element; identifying visual level features of an interface where the screenshot is located; taking a picture boundary of the screenshot as a rectangle of the outermost layer, and determining a rectangle range of the next layer; and judging whether the rectangle of the current level is the minimum rectangle containing the target element, and if the rectangle of the current level is the minimum rectangle containing the target element, performing secondary processing on the rectangle of the current level to obtain a final target element. The method supports a non-control mode to extract the target element, solves the problems that the extraction efficiency of the target element in screen pictures of different sizes is low and the target element cannot be accurately separated when interface node information is analyzed, can improve the extraction accuracy of the target element, does not need to be connected with a terminal device for operation, and improves the extraction efficiency of the target element. And the position of the target element on the screenshot picture of any size can be flexibly determined without analyzing the information of the UI node.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and information technology, and in particular to a method and system for extracting elements based on a terminal screen shot image. Background Art

[0002] In terminal device applications, screen sizes and resolutions vary, and adapting the screen becomes a technical challenge in extracting target elements. Extracting the required content from screenshots of different sizes is more challenging than development. On different devices, the size and position of the same element on the user interface may be different. Even on the same device, triggering different conditions may cause the position and size of the same element in the screenshot to change. Therefore, in the scenario where only screenshots can be obtained, it is more difficult to extract the target element.

[0003] The existing method of extracting the target elements of screenshots mainly adopts the method of extracting interface node information by on-the-spot analysis. However, when on-the-spot analysis is not possible or the number of devices is large, and it is affected by various environmental factors, the efficiency of this method will become low. In addition, some interfaces do not implement the target elements in the form of controls, but draw multiple different elements together, which makes it impossible to accurately separate the target elements when analyzing the interface node information.

[0004] In the prior art, element extraction in the terminal is generally achieved through adb, auxiliary functions, image recognition technology, and UI automated testing tools.

[0005] adb can obtain complete screenshots and UI node information and supports a variety of commands and tools, but it needs to be connected to a specific device and may encounter connection problems in the actual environment. It has limited support for elements implemented in non-control methods (such as text in images, and combinations of multiple elements implemented by drawing methods), and usually requires additional image processing.

[0006] Accessibility functions do not need to rely on physical device connections to obtain node tree information of the interface, but they also have limitations for elements implemented in non-control ways.

[0007] Image recognition technology can process any type of screenshot, but its accuracy is highly dependent on the algorithm and training data, and is also affected by image quality, resolution, etc., and image processing and recognition may require higher computing resources.

[0008] UI automation testing tools provide a rich set of API interfaces for locating, operating, and verifying UI elements, but they are also unable to handle elements that are not implemented in a control manner. They also have specific requirements for the device's operating system version and application configuration, which affects the versatility of the tool.

[0009] In view of the above problems, the present application proposes a method and system for extracting elements based on terminal screenshot images to solve the problem of extracting target elements in images. Summary of the invention

[0010] The present invention proposes the following technical solutions to address one or more technical deficiencies in the above-mentioned prior art.

[0011] Based on the first aspect of the present invention, a method for extracting elements based on a terminal screenshot image is proposed, comprising:

[0012] S1: Get a screenshot containing the complete target element;

[0013] S2: Identify the visual hierarchy features of the interface where the screenshot is located;

[0014] S3: taking the image boundary of the screenshot as the outermost rectangle and determining the range of the rectangle at the next level;

[0015] Specifically, taking the top coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as new starting points, identifying the color features of the starting points and determining whether they meet the conditions, and connecting all the starting points that meet the conditions to obtain the upper boundary of the next level rectangle;

[0016] S4: Determine whether the rectangle at the current level is the smallest rectangle that includes the target element. If it is the smallest rectangle that includes the target element, perform secondary processing on the rectangle at the current level to obtain the final target element.

[0017] Furthermore, the visual hierarchy features include dialog boxes, rectangular frames, element combination rules and picture boundaries.

[0018] Identifying visual hierarchy features can quickly capture key information, form a sense of hierarchy, distinguish the importance and priority of information, and ensure the accuracy and efficiency of information communication.

[0019] Furthermore, step S3 further includes:

[0020] Take the lowest coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the lower boundary of the next level rectangle.

[0021] Furthermore, step S3 further includes:

[0022] Take the leftmost coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the left boundary of the next level rectangle.

[0023] Furthermore, step S3 further includes:

[0024] Take the rightmost coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the right boundary of the next level rectangle.

[0025] Using the collapse algorithm to gradually reduce the boundary of the target element can improve the accuracy of target element extraction.

[0026] Furthermore, identifying the color feature of the starting point and determining whether the condition is satisfied includes identifying the pixel color value range of the starting point, the relationship between the color of the starting point and the color change of the surrounding pixels;

[0027] The relationship between the color of the starting point and the color of the surrounding pixels includes the same color, color gradient and color difference.

[0028] Furthermore, the secondary processing includes but is not limited to average segmentation and non-rectangular cutting of the rectangular image at the current level.

[0029] Secondary processing of the image can separate the target elements more accurately.

[0030] Furthermore, the target elements include but are not limited to buttons, characters, objects, symbols, rectangles of specified sizes, circles of specified sizes, and user avatars.

[0031] Based on the second aspect of the present invention, a system for extracting elements based on a terminal screenshot image is also proposed, comprising:

[0032] Get module: Get a screenshot containing the complete target element;

[0033] Recognition module: identifying the visual hierarchical features of the interface where the screenshot is located;

[0034] Determine rectangle module: take the image boundary of the screenshot as the outermost rectangle, and determine the range of the rectangle of the next level;

[0035] Specifically, taking the top coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as new starting points, identifying the color features of the starting points and determining whether they meet the conditions, and connecting all the starting points that meet the conditions to obtain the upper boundary of the next level rectangle;

[0036] Extraction module: Determine whether the rectangle at the current level is the smallest rectangle containing the target element. If it is the smallest rectangle containing the target element, perform secondary processing on the rectangle at the current level to obtain the final target element.

[0037] Based on the third aspect of the present invention, a computer program product is also proposed, comprising a computer program, and when the computer program is executed by a computer processor, the steps of any of the above methods are implemented.

[0038] The technical effect of the present invention is that: the present invention supports the extraction of target elements in a non-control manner, solves the problem of low efficiency in extracting target elements in screen images of different sizes, and the problem of being unable to accurately separate target elements when analyzing interface node information in on-the-spot analysis, can improve the accuracy of target element extraction, and does not require connection to a terminal device for operation, and does not require analysis of UI node information, so the position of the target element on a screenshot image of any size can be flexibly determined, and the template elements can also be accurately separated when analyzing interface node information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0040] Figure 1 The present invention is a general flow chart of a method for extracting elements based on a terminal screenshot image provided in accordance with an embodiment of the present invention.

[0041] Figure 2 The present invention is a schematic diagram of a method for extracting elements based on a terminal screenshot according to an embodiment of the present invention.

[0042] Figure 3 It is a framework diagram of a system for extracting elements based on terminal screenshot images provided according to an embodiment of the present invention.

[0043] Figure 4 It is a specific flow chart of a method for extracting elements based on a terminal screenshot image provided according to an embodiment of the present invention.

[0044] Figure 5 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0045] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] Figure 1 The overall flow chart of a method for extracting elements based on a terminal screenshot image of the present invention is shown, and the method comprises:

[0048] S1: Get a screenshot containing the complete target element;

[0049] S2: Identify the visual hierarchy features of the interface where the screenshot is located;

[0050] S3: taking the image boundary of the screenshot as the outermost rectangle and determining the range of the rectangle at the next level;

[0051] Specifically, taking the top coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as new starting points, identifying the color features of the starting points and determining whether they meet the conditions, and connecting all the starting points that meet the conditions to obtain the upper boundary of the next level rectangle;

[0052] S4: Determine whether the rectangle at the current level is the smallest rectangle that includes the target element. If it is the smallest rectangle that includes the target element, perform secondary processing on the rectangle at the current level to obtain the final target element.

[0053] It should be noted that the visual hierarchy features include dialog boxes, rectangular frames, element combination rules and picture boundaries.

[0054] It should be noted that identifying visual hierarchy features can quickly capture key information, form a sense of hierarchy, distinguish the importance and priority of information, and ensure the accuracy and efficiency of information communication.

[0055] It should be noted that step S3 also includes:

[0056] Take the lowest coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the lower boundary of the next level rectangle.

[0057] It should be noted that step S3 also includes:

[0058] Take the leftmost coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the left boundary of the next level rectangle.

[0059] It should be noted that step S3 also includes:

[0060] Take the rightmost coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the right boundary of the next level rectangle.

[0061] It should be noted that the accuracy of target element extraction can be improved by gradually reducing the boundary of the target element using the collapse algorithm.

[0062] It should be noted that identifying the color feature of the starting point and determining whether the condition is met includes identifying the pixel color value range of the starting point, the relationship between the color of the starting point and the color change of the surrounding pixels;

[0063] The relationship between the color of the starting point and the color of the surrounding pixels includes the same color, color gradient and color difference.

[0064] In a specific embodiment, the RGB value range of the pixel color of the starting point is specifically: R range is 0-100, G range is 100-200, and B range is 200-255.

[0065] In a specific embodiment, if the color value of the current coordinate of the starting point is consistent with the color value of the pixel points in the previous row or the color values ​​of the pixel points extending in an L-shaped route starting from the current coordinate of the starting point are the same, then the color change relationship between the color of the starting point and the surrounding pixel points is the same color;

[0066] Otherwise, if the color of the current coordinates of the starting point is pure white, and the 10 pixels in a column adjacent to the right of the current coordinates of the starting point are not all pure white, the color change relationship between the starting point and the surrounding pixels is different colors.

[0067] In a specific embodiment, the coordinates of the starting point extend 5 pixels y to the right and satisfy the formula y=ax+b with the pixel x on the left of the starting point. Then, the color change relationship between the starting point and the surrounding pixels is gradual.

[0068] In a specific embodiment, if the color R value of the pixel point adjacent to the left of the starting point is 90 and the color R value of the starting point is 99, there is a difference in the color change between the starting point and the surrounding pixels.

[0069] It should be noted that the secondary processing includes but is not limited to average segmentation and non-rectangular cutting of the rectangular image at the current level.

[0070] It should be noted that secondary processing of the image can separate the target elements more accurately.

[0071] It should be noted that the target elements include but are not limited to buttons, characters, objects, symbols, rectangles of specified sizes, circles of specified sizes, and user avatars.

[0072] In a specific embodiment, the rectangular image of the current layer is evenly divided into 3 columns and 2 rows, and a circle or a rounded rectangle of a specified size is cut out from each of the divided images, and corresponding shapes are cut out according to the image contour.

[0073] In a specific embodiment, the image boundary of the screenshot is used as the outermost rectangle, and the range of the rectangle of the next level is determined; the top, bottom, leftmost and rightmost coordinate points of the rectangle where the target element is located are used as starting points, all the starting points are moved one pixel toward the target element as new starting points, the color features of the starting points are identified and it is determined whether conditions are met, all the starting points that meet the conditions are connected to obtain the four boundaries of the rectangle of the next level, and it is determined whether the rectangle of the current level contains the target element (such as Figure 2 The minimum rectangle of element 1 and element 2 in .

[0074] It should be noted that the present invention supports non-control methods to extract target elements, solves the problem of low efficiency in extracting target elements in screen images of different sizes, and the inability to accurately separate target elements when analyzing interface node information in on-the-spot analysis, can improve the accuracy of target element extraction, and does not require connection to a terminal device for operation. It does not require analysis of UI node information, and can flexibly determine the position of target elements on screenshot images of any size, and can also accurately separate template elements when performing interface node information analysis.

[0075] Reference below Figure 3 , Figure 3 The framework diagram of a system for extracting elements based on terminal screenshot images is shown, including an acquisition module a, a recognition module b, a rectangle determination module c and an extraction module d.

[0076] In a specific embodiment, the acquisition module a is configured to acquire a screenshot containing a complete target element.

[0077] In a specific embodiment, the identification module b is configured to: identify the visual hierarchy features of the interface where the screenshot is located.

[0078] In a specific embodiment, the rectangle determination module c is configured to: take the image boundary of the screenshot as the outermost rectangle, and determine the rectangle range of the next level.

[0079] In a specific embodiment, the extraction module d is configured to: determine whether the rectangle of the current level is the smallest rectangle containing the target element; if it is the smallest rectangle containing the target element, perform secondary processing on the rectangle of the current level to obtain the final target element.

[0080] In a specific embodiment, a system for extracting elements based on a terminal screen shot image is based on Figure 4 The process shown is used to extract elements. The specific steps are as follows:

[0081] Identify visual hierarchical features of the interface where the screenshot is located in the screenshot containing the complete target element, then use the image boundary of the screenshot as the outermost rectangle, and determine the range of the rectangle of the next level;

[0082] Specifically, taking the top coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as new starting points, identifying the color features of the starting points and determining whether they meet the conditions, and connecting all the starting points that meet the conditions to obtain the upper boundary of the next level rectangle;

[0083] Taking the lowest coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as the new starting point, identifying the color feature of the starting point and judging whether it meets the conditions, connecting all the starting points that meet the conditions to obtain the lower boundary of the rectangle at the next level;

[0084] Taking the leftmost coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as the new starting point, identifying the color feature of the starting point and judging whether it meets the conditions, connecting all the starting points that meet the conditions to obtain the left boundary of the next level rectangle;

[0085] Taking the rightmost coordinate point of the rectangle where the target element is located as the starting point, moving all the starting points one pixel toward the target element as the new starting point, identifying the color feature of the starting point and judging whether it meets the conditions, connecting all the starting points that meet the conditions to obtain the right boundary of the next level rectangle;

[0086] Identifying the color feature of the starting point and determining whether the condition is met includes identifying the pixel color value range of the starting point, the color change relationship between the starting point and the surrounding pixel points, including the same color, color gradient and color difference;

[0087] Determine whether the rectangle of the current level is the minimum rectangle containing the target element. If it is the minimum rectangle containing the target element, perform secondary processing on the rectangle of the current level, including but not limited to average segmentation and non-rectangular cutting of the rectangular image of the current level, to obtain the final target element.

[0088] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0089] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0090] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0092] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0094] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0095] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: obtains a screenshot containing a complete target element; identifies the visual hierarchy features of the interface where the screenshot is located; uses the image boundary of the screenshot as the outermost rectangle, and determines the range of the rectangle of the next level; specifically, takes the top coordinate point of the rectangle where the target element is located as the starting point, moves all the starting points one pixel toward the target element as the new starting point, identifies the color features of the starting point and determines whether the conditions are met, connects all the starting points that meet the conditions to obtain the upper boundary of the rectangle of the next level; determines whether the rectangle of the current level is the minimum rectangle containing the target element, and if it is the minimum rectangle containing the target element, performs secondary processing on the rectangle of the current level to obtain the final target element.

[0096] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0097] Finally, it should be noted that the above embodiments are only intended to illustrate rather than limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for extracting elements based on a terminal screenshot image, characterized in that: include: S1: Get a screenshot containing the complete target element; S2: Identify the visual hierarchical features of the interface where the screenshot is located; S3: taking the image boundary of the screenshot as the outermost rectangle and determining the range of the rectangle at the next level; Specifically, the coordinate point at the top of the rectangle where the target element is located is taken as the starting point, all the starting points are moved one pixel toward the target element as new starting points, the color features of the starting points are identified and it is determined whether the conditions are met, and all the starting points that meet the conditions are connected to obtain the upper boundary of the next level rectangle; S4: Determine whether the rectangle at the current level is the smallest rectangle that includes the target element. If it is the smallest rectangle that includes the target element, perform secondary processing on the rectangle at the current level to obtain the final target element.

2. The method according to claim 1, characterized in that The visual hierarchy features include dialog boxes, rectangular frames, element combination rules and picture boundaries.

3. The method according to claim 1, characterized in that Step S3 also includes: Take the bottom coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the lower boundary of the next level rectangle.

4. The method according to claim 1, characterized in that: Step S3 also includes: Take the leftmost coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the left boundary of the next level rectangle.

5. The method according to claim 1, characterized in that Step S3 also includes: Take the rightmost coordinate point of the rectangle where the target element is located as the starting point, move all the starting points one pixel toward the target element as the new starting point, identify the color features of the starting point and determine whether the conditions are met, and connect all the starting points that meet the conditions to obtain the right boundary of the next level rectangle.

6. The method according to claim 1, 3, 4 or 5, characterized in that: Identifying the color feature of the starting point and determining whether the condition is met includes identifying the pixel color value range of the starting point and the relationship between the color of the starting point and the color change of the surrounding pixels; The relationship between the color of the starting point and the color of the surrounding pixels includes the same color, color gradient and color difference.

7. The method according to claim 1, characterized in that The secondary processing includes but is not limited to average segmentation and non-rectangular cutting of the rectangular image at the current level.

8. The method according to claim 1, characterized in that The target elements include but are not limited to buttons, characters, objects, symbols, rectangles of specified sizes, circles of specified sizes, and user avatars.

9. A system for extracting elements based on terminal screenshot images, characterized in that: include: Get module: Get a screenshot containing the complete target element; Recognition module: identifying the visual hierarchical features of the interface where the screenshot is located; Determine rectangle module: take the image boundary of the screenshot as the outermost rectangle, and determine the range of the rectangle at the next level; Specifically, the coordinate point at the top of the rectangle where the target element is located is taken as the starting point, all the starting points are moved one pixel toward the target element as new starting points, the color features of the starting points are identified and it is determined whether the conditions are met, and all the starting points that meet the conditions are connected to obtain the upper boundary of the next level rectangle; Extraction module: Determine whether the rectangle at the current level is the smallest rectangle containing the target element. If it is the smallest rectangle containing the target element, perform secondary processing on the rectangle at the current level to obtain the final target element.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a computer processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images