Automatic interface clicking method and device, equipment and storage medium
By comprehensively considering the outline shape and color features of control icons in the interface image for matching, the problem of inaccurate target control icon recognition in the prior art is solved, and efficient and accurate automatic operation is achieved in complex interface environments.
Patent Information
- Application Number
- CN202511671079.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies typically rely on a single dimension of features to determine target control icons in interface images, which is insufficient to handle complex element environments. This can lead to automatic click operations missing the correct target object, causing erroneous operations and hindering the efficient and accurate completion of automatic operation tasks.
By acquiring the interface image of the display device, performing image recognition processing, obtaining the control icon to be matched, and matching it with the predefined baseline control icon, taking into account the outline shape and color characteristics of the icon, determining the target control icon and its interactive position, controlling the cursor movement and simulating click operation.
Accurately identify target control icons in complex interface environments to avoid accidental operations and achieve efficient and accurate automatic operation.
Smart Images
Figure CN121635741A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computers, and in particular to an interface automatic clicking method and device, equipment and a storage medium. BACKGROUND
[0002] With the development of digital interaction technology, automatic clicking operation technology has been widely applied in the fields of automatic testing, process automation, and auxiliary operation, for example, in software automatic testing, repeated operation verification can be completed by automatically controlling the interface instead of manual operation.
[0003] However, interface elements are often complex and diverse, and different control icons may have similarities in shape, color, etc. When the target control icon to be executed for clicking operation is determined in the interface image in the prior art, only a single dimension of features is usually used for judgment, for example, whether the approximate shape of the control icon is similar for matching. This matching method is difficult to cope with complex element environment in the interface, and when the control icon is slightly deformed or interfered by the background element, the target control icon cannot be accurately identified, the automatic clicking operation loses the correct target object, and further causes misoperation, resulting in the inability to efficiently and accurately complete the automatic operation task in the complex interface environment. SUMMARY
[0004] The present application provides an interface automatic clicking method, device, equipment and storage medium, which solves the problem that the target control icon cannot be accurately identified in the prior art, the automatic clicking operation loses the correct target object, and further causes misoperation, resulting in the inability to efficiently and accurately complete the automatic operation task in the complex interface environment, improves the accuracy of identifying the target control icon, and can efficiently and accurately complete the automatic operation task in the complex interface environment.
[0005] In a first aspect, the present application provides an interface automatic clicking method, comprising: obtaining an interface image currently displayed by a display device, performing image recognition processing on the interface image to obtain at least one control icon as a to-be-matched control icon; obtaining a reference control icon of the interface image to be executed for clicking operation defined in advance, matching each to-be-matched control icon with the reference control icon, determining a target control icon in the plurality of to-be-matched control icons based on the matching result, and a corresponding interactive position of the target control icon in the interface image; controlling a cursor to move to the interactive position and simulate the execution of the corresponding clicking operation.
[0006] Optionally, the matching of each to-be-matched control icon with the reference control icon and the determination of the target control icon in the plurality of to-be-matched control icons based on the matching result comprise: determining a first contour line of each of the to-be-matched control icons, and determining first feature information based on the first contour line, wherein the first feature information comprises first inflection points and a number of the first inflection points; determining a second contour line of the reference control icon, and determining second feature information based on the second contour line, wherein the second feature information comprises second inflection points and a number of the second inflection points; matching the first feature information and the second feature information to determine a candidate control icon; obtaining a color histogram of the candidate control icon and a color histogram of the reference control icon, and determining a target control icon according to the color histogram of the candidate control icon and the color histogram of the reference control icon.
[0007] Optionally, the first feature information is determined based on the first contour line, and the determining comprises: calculating curvature values of points on the first contour line, regarding points with curvature values greater than a preset threshold as the first inflection points, and determining the number of the first inflection points; the second feature information is determined based on the second contour line, and the determining comprises: calculating curvature values of points on the second contour line, regarding points with curvature values greater than a preset threshold as the second inflection points, and determining the number of the second inflection points.
[0008] Optionally, the target control icon is determined to correspond to an interactable position in the interface image, and the determining comprises: determining a minimum circumscribed rectangle of the target control icon; determining four boundary coordinate points of the minimum circumscribed rectangle in a coordinate system of the interface image, and determining a region range surrounded by the minimum circumscribed rectangle based on the four boundary coordinate points; regarding the region range surrounded by the minimum circumscribed rectangle as the interactable position corresponding to the target control icon.
[0009] Optionally, the control cursor is moved to the interactable position, and the moving comprises: obtaining a current coordinate point of the cursor in a coordinate system of the interface image; randomly generating a target coordinate point in a region range of a minimum circumscribed rectangle of the interactable position based on the region range; generating a moving path from the current coordinate point to the target coordinate point according to the current coordinate point and the target coordinate point; generating a moving control instruction of the cursor based on the moving path, so as to control the cursor to move to the interactable position.
[0010] Optionally, the simulating performing the corresponding click operation comprises: determining a corresponding click operation mode of the reference control icon preset; generating a corresponding click operation instruction based on the click operation mode; in a case where it is detected that the cursor moves to a corresponding interactive position of the target control icon, executing the click operation instruction to simulate performing the corresponding click operation.
[0011] Optionally, after the simulating performing the corresponding click operation, the method further comprises: obtaining execution state information of the target control icon after simulating performing the corresponding click operation; judging whether the target control icon successfully responds to the click operation based on the execution state information.
[0012] In a second aspect, the present application provides an interface automatic clicking device, comprising: a recognition module configured to obtain an interface image currently displayed by a display device, perform image recognition processing on the interface image, and obtain at least one control icon as a to-be-matched control icon; a matching module configured to obtain a reference control icon of the interface image for which a click operation is to be performed, match each to-be-matched control icon with the reference control icon, determine a target control icon from the to-be-matched control icons based on a matching result, and determine a corresponding interactive position of the target control icon in the interface image; a clicking module configured to control a cursor to move to the interactive position and simulate performing a corresponding click operation.
[0013] In a third aspect, the present application provides an interface automatic clicking device, comprising: one or more processors; a memory storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the interface automatic clicking method according to the first aspect.
[0014] In a fourth aspect, the present application provides a storage medium containing computer executable instructions, when the computer executable instructions are executed by a computer processor, the computer executable instructions are used to execute the interface automatic clicking method according to the first aspect.
[0015] In summary, compared with the prior art, the technical solution provided by the present application brings at least the following effective effects: In the present application, by acquiring the interface image currently displayed by the display device, image recognition processing is performed on the interface image to obtain at least one control icon as a to-be-matched control icon; a reference control icon on which a click operation needs to be performed is acquired in advance for the interface image, each to-be-matched control icon is matched with the reference control icon, a target control icon is determined in the plurality of to-be-matched control icons based on the matching result, and a corresponding interactive position of the target control icon in the interface image is determined; the cursor is controlled to move to the interactive position, and the corresponding click operation is simulated to be performed. In the above technical means, each to-be-matched control icon is matched with the reference control icon one by one, and the matching manner in the present application comprehensively considers the contour shape and color of the icon, so that the target control icon can be accurately identified in a complex element environment in the interface, and the misoperation caused by the automatic click operation losing the correct target object is avoided, and the automatic operation task can be efficiently and accurately completed. BRIEF DESCRIPTION OF DRAWINGS
[0016] Figure 1 is a flowchart of an interface automatic clicking method provided by an embodiment of the present application; Figure 2 is a flowchart of determining a target control icon provided by an embodiment of the present application; Figure 3 is a flowchart of determining a corresponding interactive position of the target control icon in the interface image provided by an embodiment of the present application; Figure 4 is a flowchart of controlling the cursor to move to the interactive position provided by an embodiment of the present application; Figure 5 is a flowchart of simulating to perform the corresponding click operation provided by an embodiment of the present application; Figure 6 is a flowchart of judging whether the target control icon successfully responds to the click operation provided by an embodiment of the present application; Figure 7 is a structural schematic diagram of an interface automatic clicking device provided by an embodiment of the present application; Figure 8 is a structural schematic diagram of an interface automatic clicking device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0017] In order to make the purposes, technical solutions and advantages of the present application clearer, the following further describes the specific embodiments of the present application with reference to the drawings. It should be understood that the specific embodiments described herein are merely intended to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all. Before discussing the example embodiments in more detail, it should be mentioned that some example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when the operations are completed, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, etc.
[0018] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not intended to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.
[0019] The prior art determines the target control icon to be executed in the interface image, usually only through a single dimension of features, for example, by matching whether the approximate shape of the control icon is similar. This matching method is difficult to cope with complex element environment in the interface, when the control icon is slightly deformed or interfered by the background element, the target control icon cannot be accurately identified, the automatic click operation loses the correct target object, and further causes misoperation, resulting in that the automatic operation task cannot be efficiently and accurately completed in the complex interface environment.
[0020] In order to solve the technical problems that the target control icon cannot be accurately identified in the prior art, the automatic click operation loses the correct target object, and further causes misoperation, resulting in that the automatic operation task cannot be efficiently and accurately completed in the complex interface environment, the embodiments of the present application provide an interface automatic click method, by one-to-one matching each to-be-matched control icon with a reference control icon, and the matching method in the present application considers the contour shape and color of the icon, the target control icon can be accurately identified in the complex element environment in the interface, the misoperation caused by the automatic click operation losing the correct target object is avoided, and the automatic operation task can be efficiently and accurately completed.
[0021] The interface automatic clicking method provided in the embodiment can be executed by an interface automatic clicking device, which can be implemented in software and / or hardware, and can be composed of two or more physical entities or one physical entity.
[0022] The interface automatic clicking device is installed with at least one type of operating system, including but not limited to an Android system, a Linux system, and a Windows system. The interface automatic clicking device can install at least one application based on the operating system, which can be an application provided by the operating system or an application downloaded from a third-party device or server. In the embodiment, the interface automatic clicking device has at least one application that can execute the interface automatic clicking method.
[0023] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0024] Figure 1 A flowchart of an interface automatic clicking method provided by an embodiment of the present application is given. Please refer to Figure 1 The interface automatic clicking method specifically includes: S110, obtaining an interface image currently displayed by a display device, performing image recognition processing on the interface image, and obtaining at least one control icon as a to-be-matched control icon.
[0025] The display device refers to an electronic device capable of presenting a visual interface, which converts digital information into image content that can be visually perceived by a user through a screen or other display components. Exemplarily, the display device includes but is not limited to a display of a desktop computer, a notebook computer, a tablet computer, etc.
[0026] The control icon refers to a visual element in the interface presented by the display device, which is used to realize a specific interactive function and is an object for operating and interacting with the interface, such as a confirm button control icon, a close button control icon, a submit button control icon, an arrow control icon, a menu control icon, etc. in a certain interface of the display device.
[0027] In the embodiments of the present application, the obtained interface image is complete image data containing all visual elements in the interface, which can be in a common image format such as JPEG or PNG.
[0028] Further, since the interface image can have noise introduced by the display device itself or the acquisition process, the obtained interface image needs to be denoised. A median filter algorithm or a Gaussian filter algorithm can be used to remove noise to reduce noise interference in the subsequent recognition.
[0029] Exemplarily, in the embodiments of the present application, a target detection algorithm (such as a YOLO algorithm based on deep learning, an SSD algorithm, or a detection algorithm based on feature matching) can be used to analyze the denoised interface image, that is, by identifying elements with specific visual features (such as independent graphic boundaries and regions with obvious distinction from the surrounding background) in the image, the regions belonging to the control icons in the interface image are identified from the overall image. In the identification process, non-interactive elements (such as background pictures and decorative patterns) in the interface are excluded according to the features of the control icons, such as closure and independence, and finally at least one region meeting the control icon features is determined, and the image part corresponding to the region is taken as the control icon to be matched.
[0030] S120, obtaining a reference control icon in the interface image that needs to perform a click operation, matching each of the control icons to be matched with the reference control icon, determining a target control icon in the plurality of control icons to be matched based on the matching result, and a corresponding interactive position of the target control icon in the interface image.
[0031] Specifically, the user will predefine the reference control icon in each interface that needs to perform a click operation and the click operation mode corresponding to each reference control icon, such as single click or double click.
[0032] The reference control icon refers to a control icon template that the user sets in advance in a specific interface that needs to perform a click operation, which is a reference standard for subsequent matching of the control icon to be matched.
[0033] Optionally, Figure 2 is a flowchart for determining a target control icon provided by the embodiments of the present application. As shown in Figure 2 the step of determining a target control icon specifically includes S1201-S1204: S1201, determining a first contour line of each of the control icons to be matched, and determining first feature information based on the first contour line, wherein the first feature information includes a first inflection point and a number of first inflection points.
[0034] The contour line refers to a continuous curve or a broken line of the edge of the control icon, which is the boundary contour of the shape of the control icon and can clearly outline the geometric shape and boundary range of the control icon. Exemplarily, the edge information of the control icon can be extracted by performing edge detection processing (such as a Canny edge detection algorithm) on the image data of each control icon to be matched, an edge pixel is found as a starting point in the edge information, and then the direction is tracked along the edge pixel until the starting point is returned, and a closed contour line is obtained, that is, the contour line of the control icon.
[0035] The inflection point refers to a point on the contour line where the direction of the curve changes significantly. Specifically, it is a point on the contour line with a large curvature, reflecting the key turning position of the shape of the control icon. Optionally, the specific steps of determining the first feature information based on the first contour line in the embodiments of the present application include: The curvature values of the points on the first contour line are calculated, and the points with curvature values greater than a preset threshold are taken as the first inflection points, and the number of the first inflection points is determined. The curvature value is a physical quantity describing the degree of bending of the curve. The greater the curvature value, the greater the degree of bending of the curve at the point. For example, in the contour line of a rectangular control icon, the four corner points of the rectangular control icon are inflection points, and their curvature values are significantly higher than those of other points on the edges.
[0036] For example, first, the pixel coordinates (x, y) of all points on the contour line are extracted, and the coordinate points are arranged in order according to their order on the contour line to form an ordered point set. Second, for each point on the contour line, a curve curvature calculation method (such as a method based on circle fitting) is used to calculate the curvature value of each point. A preset curvature threshold is set, which can be set and adjusted according to the actual application scenario. The curvature value of each point calculated is compared with the preset threshold, and the point with a curvature value greater than the preset threshold is determined as an inflection point.
[0037] S1202, determine the second contour line of the reference control icon, and determine the second feature information based on the second contour line, wherein the second feature information includes the second inflection point and the number of the second inflection point.
[0038] Further, the curvature values of the points on the second contour line are calculated, and the points with curvature values greater than a preset threshold are taken as the second inflection points, and the number of the second inflection points is determined.
[0039] S1203, match the first feature information and the second feature information to determine the candidate control icon.
[0040] In the embodiments of the present application, the inflection points of the control icon and the number of the inflection points can more accurately determine the shape essence of the control icon.
[0041] S1204, obtain the color histogram of the candidate control icon and the color histogram of the reference control icon, and determine the target control icon according to the color histogram of the candidate control icon and the color histogram of the reference control icon.
[0042] The color histogram is a kind of method for describing the color distribution characteristics of the image. By quantifying the color information of the image, the color distribution is visually displayed in the form of a histogram.
[0043] Further, after obtaining the color histogram of the candidate control icon and the reference control icon, the similarity between the two is calculated to determine the color matching degree of the candidate control icon and the reference control icon.
[0044] By color histogram comparison, control icons with similar shapes but different colors can be effectively distinguished (for example, the confirm button control icon (green) and the cancel button control icon (red) are both rectangles).
[0045] In the embodiments of the present application, the candidate control icon is first determined through the inflection point and the number of inflection points of the control icon, and then the target control icon matching the reference control icon is determined from the candidate control icon in terms of color, that is, the target control icon is screened from two dimensions, which improves the accuracy of identifying the target control icon in a complex interface.
[0046] In the embodiments of the present application, the interactive position refers to the specific area range of the control icon in the interface image that can be interacted, the area that can trigger the function corresponding to the control icon, and can also be understood as the target area to which the cursor needs to be moved. Only when the clicking operation is performed in this area, can the target control icon be ensured to respond correctly (for example, the button is clicked to trigger the jump, the input box is clicked to activate the input function, etc.).
[0047] Optionally, Figure 3 is a flowchart provided by the embodiments of the present application for determining the corresponding interactive position of the target control icon in the interface image. As shown in Figure 3 the step of determining the corresponding interactive position of the target control icon in the interface image specifically includes S1205-S1207: S1205, determining the minimum circumscribed rectangle of the target control icon.
[0048] The minimum circumscribed rectangle refers to the rectangle that can completely enclose the target control icon and has the smallest area. For example, the maximum and minimum horizontal and vertical coordinate values of the target control icon in the interface image can be determined by calculating the coordinates of all pixel points of the target control icon, so as to obtain the boundary range of the target control icon. Specifically, the horizontal coordinates of the leftmost and rightmost pixel points of the target control icon in the horizontal direction and the vertical coordinates of the topmost and bottommost pixel points of the target control icon in the vertical direction are found. Based on the above four boundary coordinate points, a minimum rectangular area that can completely enclose the target control icon is constructed, which is the minimum circumscribed rectangle of the target control icon.
[0049] S1206, determining the four boundary coordinate points of the minimum circumscribed rectangle in the coordinate system of the interface image, and determining the area range enclosed by the minimum circumscribed rectangle based on the four boundary coordinate points.
[0050] Specifically, four boundary coordinate points of the minimum bounding rectangle, i.e., the coordinate points of the top-left corner, the top-right corner, the bottom-right corner and the bottom-left corner, are extracted; and the area range surrounded by the minimum bounding rectangle is calculated according to the four boundary coordinate points of the minimum bounding rectangle.
[0051] S1207, taking the area range surrounded by the minimum bounding rectangle as the interactive position corresponding to the target control icon.
[0052] S130, controlling the cursor to move to the interactive position to simulate performing a corresponding click operation.
[0053] The cursor is a pointer displayed on the screen and can be an arrow, a cross or an icon of other shapes. The click operation can be understood as a trigger operation performed on a control icon on the interface, and the purpose is to activate the function or response logic corresponding to the control icon.
[0054] Optionally, Figure 4 is a flowchart of the embodiment provided by the present application for controlling the cursor to move to the interactive position. As shown in Figure 4 , the step of controlling the cursor to move to the interactive position specifically includes S1301-S1304: S1301, obtaining a current coordinate point of the cursor in a coordinate system of the interface image.
[0055] S1302, randomly generating a target coordinate point in the area range of the minimum bounding rectangle corresponding to the interactive position.
[0056] Specifically, the X-axis coordinate and the Y-axis coordinate of the minimum bounding rectangle are taken as the constraint conditions for random generation, and then a pseudo-random number generation algorithm is used to generate a random number on the X-axis and the Y-axis, respectively. The random numbers generated on the X-axis and the Y-axis are combined, and a target coordinate point in the area range of the minimum bounding rectangle is obtained.
[0057] In the embodiment of the present application, the target coordinate point in the area range is randomly generated, which can make the automatic click operation closer to the behavior habits of real users and improve the authenticity of the automatic operation. For example, in some scenarios that need to simulate manual operation, such as automated testing or auxiliary operation, randomness can prevent the tested system or the auxiliary object from easily identifying that the operation is performed by an automated program, thereby improving the accuracy of the test result and the effectiveness of the auxiliary operation.
[0058] S1303, generating a moving path from the current coordinate point to the target coordinate point according to the current coordinate point and the target coordinate point.
[0059] For example, according to the current coordinate point and the target coordinate point, a straight line path can be generated, that is, a straight line distance between the current coordinate point and the target coordinate point is calculated. In order to simulate a more natural cursor moving track, a curved path planning can also be used, for example, a Bezier curve is used to generate a smooth curve from the current coordinate point to the target coordinate point. By setting the control points of the Bezier curve, the shape and bending degree of the curve can be controlled, so that the movement of the cursor is more in line with the user's daily operation habits, and the stiffness caused by the straight line movement can be avoided.
[0060] S1304, based on the moving path, a moving control instruction of the cursor is generated to control the cursor to move to the interactable position.
[0061] Specifically, according to the generated moving path, the path information is converted into a cursor moving control instruction; the control instruction is sent to the operating system or the input device driver in the form of a pulse signal or a coordinate increment, so as to control the cursor to move according to the predetermined path.
[0062] Optionally, Figure 5 is a flowchart of simulating the execution of the corresponding click operation provided by the embodiments of the present application. As shown in the figure, the step of simulating the execution of the corresponding click operation specifically includes S1305-S1307: Figure 5 S1305, a corresponding click operation mode preset for the reference control icon is determined.
[0063] The click operation mode refers to the way of interacting with the control icon, for example, single click or double click, etc. In the embodiments of the present application, when the user defines the reference control icon, it will be bound with a specific click operation mode in advance, and stored in the system database. After the target control icon is determined, the click operation mode preset for the reference control icon matched with the target control icon in the database is searched. For example, if the reference control icon matched with the target control icon is a login button, and its preset operation mode is single click, it is determined that the single click operation needs to be executed.
[0064] S1306, a corresponding click operation instruction is generated based on the click operation mode.
[0065] S1307, in the case that the cursor moves to the interactable position corresponding to the target control icon, the click operation instruction is executed to simulate the execution of the corresponding click operation.
[0066] In the embodiments of the present application, after simulating the execution of the corresponding click operation, it further includes judging whether the target control icon successfully responds to the click operation. Optionally, Figure 6 is a flowchart of judging whether the target control icon successfully responds to the click operation provided by the embodiments of the present application. As shown in the figure, the step of judging whether the target control icon successfully responds to the click operation specifically includes S1308-S1310: Figure 6 As shown, the step of judging whether the target control icon successfully responds to the click operation specifically includes S1308-S1309: S1308, obtaining execution state information of the target control icon after simulating execution of the corresponding click operation.
[0067] The execution state information refers to various information presented by the target control icon after being simulated to execute the click operation, through interface feedback, system log or internal state change of the application program, and is used to reflect the response of the control to the click operation.
[0068] S1309, judging whether the target control icon successfully responds to the click operation based on the execution state information.
[0069] In summary, the interface automatic clicking method provided by the embodiment of the present application obtains the interface image currently displayed by the display device, performs image recognition processing on the interface image, and obtains at least one control icon as a to-be-matched control icon; obtains a reference control icon of the interface image that needs to execute a click operation, matches each to-be-matched control icon with the reference control icon, determines a target control icon in the plurality of to-be-matched control icons based on a matching result, and determines a corresponding interactive position of the target control icon in the interface image; controls the cursor to move to the interactive position and simulates execution of the corresponding click operation. In the above technical means, each to-be-matched control icon is matched with the reference control icon one by one, and the matching manner in the present application comprehensively considers the contour shape and color of the icon, so that the target control icon can be accurately identified in a complex element environment in the interface, the misoperation caused by the loss of the correct target object in the automatic click operation is avoided, and the automatic operation task can be efficiently and accurately completed.
[0070] Figure 7 A structural schematic diagram of an interface automatic clicking device provided by the embodiment of the present application is shown in FIG. 2. Figure 7 The interface automatic clicking device provided by the embodiment of the present application specifically includes an identification module 21, a matching module 22 and a clicking module 23.
[0071] The identification module 21 is used to obtain an interface image currently displayed by a display device, perform image recognition processing on the interface image, and obtain at least one control icon as a to-be-matched control icon. The matching module 22 is used to obtain a reference control icon of the interface image that needs to execute a click operation, match each to-be-matched control icon with the reference control icon, determine a target control icon in the plurality of to-be-matched control icons based on a matching result, and determine a corresponding interactive position of the target control icon in the interface image. The clicking module 23 is configured to control the cursor to move to the interactive position and simulate performing a corresponding clicking operation.
[0072] On the basis of the above embodiment, the matching module 22 comprises a target icon determining unit, and the target icon determining unit comprises: a first feature information determining subunit configured to determine a first contour line of each of the to-be-matched control icons, determine first feature information based on the first contour line, and wherein the first feature information comprises a first inflection point and a number of the first inflection points; a second feature information determining subunit configured to determine a second contour line of the reference control icon, determine second feature information based on the second contour line, and wherein the second feature information comprises a second inflection point and a number of the second inflection points; a feature matching subunit configured to match the first feature information and the second feature information to determine a candidate control icon; a target icon determining subunit configured to acquire a color histogram of the candidate control icon and a color histogram of the reference control icon, and determine a target control icon based on the color histogram of the candidate control icon and the color histogram of the reference control icon.
[0073] On the basis of the above embodiment, the first feature information determining subunit comprises: calculating a curvature value of each point on the first contour line, regarding a point with a curvature value greater than a preset threshold as the first inflection point, and determining the number of the first inflection points; On the basis of the above embodiment, the second feature information determining subunit comprises: calculating a curvature value of each point on the second contour line, regarding a point with a curvature value greater than a preset threshold as the second inflection point, and determining the number of the second inflection points.
[0074] On the basis of the above embodiment, the matching module 22 comprises an interactive position determining unit, and the interactive position determining unit comprises: a rectangle determining subunit configured to determine a minimum circumscribed rectangle of the target control icon; a range determining subunit configured to determine four boundary coordinate points of the minimum circumscribed rectangle in a coordinate system of the interface image, and determine a range of an area surrounded by the minimum circumscribed rectangle based on the four boundary coordinate points; an interactive position determining subunit configured to take the range of the area surrounded by the minimum circumscribed rectangle as an interactive position corresponding to the target control icon.
[0075] On the basis of the above embodiment, the clicking module 23 comprises a moving unit, and the moving unit comprises: A current coordinate obtaining subunit is configured to obtain a current coordinate point of the cursor in a coordinate system of the interface image; A target coordinate generating subunit is configured to randomly generate a target coordinate point within a region range of the minimum circumscribed rectangle corresponding to the interactable position based on the region range; A path generating subunit is configured to generate a moving path from the current coordinate point to the target coordinate point according to the current coordinate point and the target coordinate point; A moving subunit is configured to generate a moving control instruction of the cursor based on the moving path to control the cursor to move to the interactable position.
[0076] On the basis of the above embodiment, the clicking module 23 comprises a clicking unit, and the clicking unit comprises: A mode determining subunit is configured to determine a preset corresponding clicking operation mode of the reference control icon; An instruction generating subunit is configured to generate a corresponding clicking operation instruction based on the clicking operation mode; A clicking subunit is configured to execute the clicking operation instruction to simulate the execution of the corresponding clicking operation in a case where the cursor is detected to move to the interactable position corresponding to the target control icon.
[0077] On the basis of the above embodiment, the method further comprises: Obtaining execution state information of the target control icon after the target control icon simulates the execution of the corresponding clicking operation; Judging whether the target control icon successfully responds to the clicking operation based on the execution state information.
[0078] To sum up, the interface automatic clicking device provided by the embodiment of the present application obtains an interface image currently displayed by a display device, performs image recognition processing on the interface image, and obtains at least one control icon as a to-be-matched control icon. A reference control icon on which a clicking operation needs to be performed in the interface image is defined in advance. Each to-be-matched control icon is matched with the reference control icon. A target control icon is determined in the plurality of to-be-matched control icons based on a matching result, and a corresponding interactable position of the target control icon in the interface image is determined. The cursor is controlled to move to the interactable position, and a corresponding clicking operation is simulated. In the above technical means, each to-be-matched control icon is matched with the reference control icon one by one, and the matching manner in the present application comprehensively considers the contour shape and color of the icon. The target control icon can be accurately identified in a complex element environment in the interface, and misoperation caused by the loss of a correct target object in automatic clicking operation is avoided. The automatic operation task can be efficiently and accurately completed.
[0079] The interface automatic clicking device provided by the embodiments of the present application can be used to execute the interface automatic clicking method provided by the embodiments of the present application, and has corresponding functions and beneficial effects.
[0080] Figure 8 is a structural schematic diagram of an interface automatic clicking device provided by the embodiments of the present application, referring to Figure 8 The interface automatic clicking device includes a processor 31, a memory 32, a communication device 33, an input device 34 and an output device 35. The number of the processor 31 in the interface automatic clicking device can be one or more, and the number of the memory 32 in the interface automatic clicking device can be one or more. The processor 31, the memory 32, the communication device 33, the input device 34 and the output device 35 of the interface automatic clicking device can be connected through a bus or other means.
[0081] The memory 32, as a computer readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules of the interface automatic clicking method of any embodiments of the present application (for example, the identification module 21, the matching module 22 and the clicking module 23 in the interface automatic clicking device). The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the device, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device or other non-volatile solid-state storage device. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.
[0082] The communication device 33 is used for data transmission.
[0083] The processor 31 executes various function applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned interface automatic clicking method.
[0084] The input device 34 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 35 can include a display device such as a display screen.
[0085] The interface automatic clicking device provided above can be used to execute the interface automatic clicking method provided by the embodiments of the present application, and has corresponding functions and beneficial effects.
[0086] The embodiment of the present application further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to perform an interface automatic clicking method, the interface automatic clicking method comprising: acquiring an interface image currently displayed by a display device, performing image recognition processing on the interface image to obtain at least one control icon as a to-be-matched control icon; acquiring a reference control icon of the interface image that needs to perform a clicking operation, matching each to-be-matched control icon with the reference control icon, determining a target control icon in the plurality of to-be-matched control icons based on a matching result, and a corresponding interactive position of the target control icon in the interface image; and controlling a cursor to move to the interactive position and simulate performing a corresponding clicking operation.
[0087] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include an installation medium, e.g., a CD-ROM, floppy disks, or tape device; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; or a non-volatile memory such as a magnetic medium (e.g., a hard disk or optical storage); registers, or other similar types of memory elements, etc. The storage medium can also include other types of storage medium and combinations thereof. Additionally, the storage medium can be located in a first computer system in which the programs are executed, or the storage medium can be located in a second different computer system which connects to the first computer system over a network such as the Internet. The second computer system can provide program instructions to the first computer system for execution. The term "storage medium" can include two or more storage mediums that reside in different locations, e.g., in different computer systems that are connected over a network. The storage medium can store program instructions (e.g., as an installed program) that can be executed by one or more processors.
[0088] Of course, the storage medium provided by the embodiment of the present application comprises computer executable instructions, which are not limited to the interface automatic clicking method as above, but can also perform the related operations in the interface automatic clicking method provided by any embodiment of the present application.
[0089] The interface automatic clicking device, the storage medium and the interface automatic clicking equipment provided in the above embodiments can perform the interface automatic clicking method provided by any embodiment of the present application, and the technical details not described in detail in the above embodiments can be referred to the interface automatic clicking method provided by any embodiment of the present application.
[0090] The above merely describes the preferred embodiments of the present application and the technical principles applied. The present application is not limited to the specific embodiments herein, and various obvious changes, modifications and replacements made by those skilled in the art without departing from the scope of the present application shall not be excluded. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and more other equivalent embodiments can be included without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.
Claims
1. An interface automatic clicking method, characterized in that, The method comprises the following steps: acquiring an interface image currently displayed by a display device, performing image recognition on the interface image, and obtaining at least one control icon as a to-be-matched control icon; acquiring a reference control icon of the interface image that needs to perform a click operation, matching each to-be-matched control icon with the reference control icon, determining a target control icon in the to-be-matched control icons based on a matching result, and determining a corresponding interactive position of the target control icon in the interface image; controlling a cursor to move to the interactive position and simulating a corresponding click operation.
2. The interface automatic clicking method of claim 1, wherein, The matching of each to-be-matched control icon with the reference control icon and the determination of a target control icon based on a matching result comprise the following steps: determining a first contour line of each to-be-matched control icon, determining first feature information based on the first contour line, wherein the first feature information comprises a first inflection point and a number of first inflection points; determining a second contour line of the reference control icon, determining second feature information based on the second contour line, wherein the second feature information comprises a second inflection point and a number of second inflection points; matching the first feature information and the second feature information to determine a candidate control icon; acquiring a color histogram of the candidate control icon and a color histogram of the reference control icon, and determining a target control icon based on the color histogram of the candidate control icon and the color histogram of the reference control icon.
3. The interface automatic clicking method of claim 2, wherein, The determination of first feature information based on the first contour line comprises the following steps: calculating the curvature value of each point on the first contour line, taking a point with a curvature value greater than a preset threshold as the first inflection point, and determining the number of first inflection points; The determination of second feature information based on the second contour line comprises the following steps: calculating the curvature value of each point on the second contour line, taking a point with a curvature value greater than a preset threshold as the second inflection point, and determining the number of second inflection points.
4. The interface automatic clicking method of claim 1, wherein, The determination of a corresponding interactive position of the target control icon in the interface image comprises the following steps: determining a minimum bounding rectangle of the target control icon; determining four boundary coordinate points of the minimum bounding rectangle in a coordinate system of the interface image, determining a region range surrounded by the minimum bounding rectangle based on the four boundary coordinate points; taking the region range surrounded by the minimum bounding rectangle as the corresponding interactive position of the target control icon.
5. The interface automatic clicking method of claim 4, wherein, The control of the cursor moving to the interactive position comprises the following steps: acquiring a current coordinate point of the cursor in a coordinate system of the interface image; randomly generating a target coordinate point within a region range of the minimum bounding rectangle corresponding to the interactive position based on the region range; generating a moving path from the current coordinate point to the target coordinate point according to the current coordinate point and the target coordinate point; generating a moving control instruction of the cursor based on the moving path to control the cursor to move to the interactive position.
6. The interface automatic clicking method of claim 1, wherein, The simulation of a corresponding click operation comprises the following steps: determining a preset corresponding click operation mode of the reference control icon; Generate a corresponding click operation instruction based on the click operation mode; In the case where the cursor moves to the interactive position corresponding to the target control icon, execute the click operation instruction to simulate the execution of the corresponding click operation.
7. The interface automatic clicking method of claim 1, wherein, After the simulation of the corresponding click operation, further comprising: Obtain the execution state information of the target control icon after the simulation of the corresponding click operation; Based on the execution state information, determine whether the target control icon successfully responds to the click operation.
8. An interface automatic clicking device, characterized in that, Comprise: An identification module is configured to obtain an interface image currently displayed by a display device, perform image recognition on the interface image, and obtain at least one control icon as a to-be-matched control icon; A matching module is configured to obtain a reference control icon of the interface image that needs to perform a click operation, match each to-be-matched control icon with the reference control icon, determine a target control icon from the to-be-matched control icons based on a matching result, and determine a corresponding interactive position of the target control icon in the interface image; A click module is configured to control a cursor to move to the interactive position and simulate the execution of a corresponding click operation.
9. An interface automatic clicking device, characterized by, Comprise: One or more processors; A memory storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the interface automatic click method as claimed in any one of claims 1-7.
10. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are used to execute the interface automatic click method as claimed in any one of claims 1-7.