Interface element control method and device, electronic equipment and storage medium

Abstract

The invention provides an interface element control method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an image of a current display interface if a target control instruction for a target interface element in the current display interface is detected; based on the image of the current display interface, determining a target area of a target text corresponding to the target interface element in the current display interface; determining a first position of the target icon in the target area; wherein the target icon is an icon corresponding to the target interface element; determining the first position as a target position of the target interface element; and executing a control operation corresponding to the target control instruction on the target interface element at the target position. According to the method, the interface elements in the display interface can be accurately positioned, and the corresponding operation is executed.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and more specifically, to a control method, device, electronic device and storage medium for interface elements in the field of computer application technology. Background Art

[0002] In the prior art, fixed coordinates or preset identifiers are usually relied upon to locate and operate interface elements in a display interface; however, the coordinates of interface elements may not be the same in different resolutions, different system versions or different devices; and the interface elements may be in dynamic change; if fixed coordinates or preset identifiers are used, the interface elements may be inaccurately positioned and correct control operations may not be performed.

[0003] Therefore, how to accurately locate interface elements in the display interface and perform corresponding control operations is a technical problem that needs to be solved at present. Summary of the invention

[0004] The present application provides a method, device, electronic device and storage medium for controlling interface elements. When the method detects a target control instruction, it can accurately identify the target interface element in the current display interface based on the image and target text of the current display interface, and perform the control operation corresponding to the target control instruction on the target interface element.

[0005] In a first aspect, a method for controlling an interface element is provided, the method comprising:

[0006] If a target control instruction for a target interface element in the current display interface is detected, an image of the current display interface is obtained;

[0007] Based on the image of the current display interface, determine a target area of ​​the target text corresponding to the target interface element in the current display interface;

[0008] In the target area, determining a first position of a target icon; wherein the target icon is an icon corresponding to the target interface element;

[0009] The first position is determined as the target position of the target interface element; and a control operation corresponding to the target control instruction is performed on the target interface element at the target position.

[0010] In an embodiment of the present application, if a target control instruction for a target interface element in the current display interface is detected, an image of the current display interface is obtained, and a target text corresponding to the target interface element is determined in a target area in the current display interface; and a first position of the target icon is determined in the target area; since there may be multiple icons of similar shapes in the display interface; for example, when the target control instruction is "turn on the Bluetooth switch", there may be multiple switch icons of similar shapes in the display interface; therefore, the target area where the target text is located is determined, and the first position of the target icon is determined in the target area; to avoid the inability to accurately identify the target icon corresponding to the target interface element when there are multiple icons of similar shapes in the current display interface; to ensure accurate identification of the target icon in the current display interface; that is, to accurately identify the target interface element in the current display interface; further, the first position is determined as the target position of the target interface element; and the operation corresponding to the target control instruction is performed on the target interface element at the target position; thereby being able to accurately locate the target interface element in the current display interface and accurately execute the control instruction for the target interface element.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0012] Obtaining preset first contour information;

[0013] In the target area, determining a first position of the target icon includes:

[0014] Based on the image analysis of the target area, determine the contour information and position of each icon in the target area;

[0015] Determine a target icon based on a first similarity between the contour information of each icon and the first contour information; wherein the first similarity between the contour information of the target icon and the first contour information is greater than a preset threshold;

[0016] Based on the positions of the target icon and each icon, a first position of the target icon is determined.

[0017] In a possible implementation manner, the first outline information is outline information corresponding to a preset target interface element.

[0018] In an embodiment of the present application, based on an image analysis of a target area, the contour information of each icon in the target area and the position of each icon are determined; the contour information of each icon is compared with the preset first contour information to determine the first similarity between the contour information of each icon and the first contour information; based on the first similarity, the target icon among the icons is determined; since the first contour information is the contour information corresponding to the preset target interface element; when the first similarity between the contour information of the icon and the first contour information is high, it indicates that the icon may be the target icon; therefore, the target icon among the icons is determined based on the first similarity; it is ensured that the obtained target icon is the icon corresponding to the target interface element; further, based on the position of the target icon and each icon, it is ensured that the first position of the target icon can be obtained.

[0019] In combination with the first aspect and the foregoing implementation manners, in some implementation manners of the first aspect, determining the target icon based on a first similarity between the outline information of each icon and the first outline information includes:

[0020] Based on the first similarity and the position information, a target icon is determined; wherein the position information includes the position of the target text and the position of each icon.

[0021] In an embodiment of the present application, a target icon among each icon is determined based on the first similarity and the position information; since there may be multiple icons with similar shapes in the target area; that is, there may be multiple icons with a first similarity greater than a preset threshold in the target area; therefore, based on the first similarity and the position information, the target icon is determined; ensuring that when there are multiple icons with similar shapes in the target area, the target icon corresponding to the target interface element can be determined.

[0022] In combination with the first aspect and the above implementations, in some implementations of the first aspect, determining a target icon among the icons based on the first similarity and the position information includes:

[0023] Determine a first icon among the icons whose first similarity is greater than a preset threshold;

[0024] Determining a target distance between the target text and the first icon based on the position of the target text and the position of the first icon;

[0025] If the target distance is less than the preset distance, and the first icon and the target text are aligned in a first direction, the first icon is determined as the target icon; wherein the first direction includes a horizontal direction or a vertical direction.

[0026] In an embodiment of the present application, a first icon among the icons whose first similarity is greater than a preset threshold is determined; and a target distance between the target text and the first icon is determined; if the target distance is less than the preset distance, and the first icon and the target text are aligned in the horizontal direction or the vertical direction, the first icon is determined as the target icon; because in the display interface, the text corresponding to the interface element and the icon corresponding to the interface element are usually close to each other and are aligned in the horizontal direction or the vertical direction; therefore, if the target distance between the first icon and the target text is less than the preset distance, and they are aligned in the first direction (horizontally or vertically), it indicates that the first icon is the icon corresponding to the target text; that is, the first icon is the target icon.

[0027] In combination with the first aspect and the above implementations, in some implementations of the first aspect, determining a target area of ​​text corresponding to a target interface element in a current display interface includes:

[0028] Perform optical character recognition on the image of the current display interface to determine the initial area where the target text is located;

[0029] The initial area is expanded based on the first pixel value to obtain a target area.

[0030] In an embodiment of the present application, optical character recognition is performed on the image of the current display interface to determine the initial area where the target text is located; and the initial area is expanded to obtain a target area containing the target text; since the initial area is the area where the target text is located; the target area obtained by expanding the initial area includes the target text and the image area around the target text; that is, a target area containing more information is obtained, which facilitates image analysis of the target area and determines the target icon in the target area.

[0031] In combination with the first aspect and the above implementation manner, in some implementation manners of the first aspect, if the target control instruction indicates a click operation on the target interface element; performing the control operation corresponding to the target control instruction on the target interface element at the target position includes:

[0032] Determine the center point of the target interface element;

[0033] Call the system's application program interface to send click events to the center point;

[0034] Based on the click event, perform a click operation at the center point of the target interface element.

[0035] In an embodiment of the present application, if the target control instruction indicates a click operation on the target interface element, the center point of the target interface element is determined, the system's application program interface is called, and a click event is sent to the center point; and based on the click event, a click operation is performed at the center point of the target interface element; ensuring that the click operation corresponding to the target control instruction can be performed on the target interface element, so that the state of the target interface element changes.

[0036] In combination with the first aspect and the above implementations, in some implementations of the first aspect, the following further includes:

[0037] If the target text does not exist in the current display interface, and / or the target icon does not exist in the current display interface, a first operation is performed to obtain a second display interface; wherein the first operation is used to switch the current display interface;

[0038] A target position of the target interface element is determined in the second display interface.

[0039] In an embodiment of the present application, if the target text does not exist in the current display interface, or the target icon does not exist in the current display interface, the first operation is performed to obtain the second display interface, and the target position of the target interface element is determined in the second display interface; when the target interface element does not exist in the current display interface, the display interface is switched, and the target position of the target interface element is determined in the second display interface after the switch.

[0040] In a second aspect, a device for controlling an interface element is provided, the device comprising:

[0041] An acquisition module, configured to acquire an image of the current display interface if a target control instruction for a target interface element in the current display interface is detected;

[0042] A first determination module is used to determine a target area of ​​a target text corresponding to a target interface element in the current display interface based on an image of the current display interface;

[0043] A second determination module is used to determine a first position of a target icon in a target area; wherein the target icon is an icon corresponding to a target interface element;

[0044] A third determining module, used to determine the first position as a target position of a target interface element;

[0045] The execution module is used to execute the control operation corresponding to the target control instruction on the target interface element at the target position.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition module is further used to: acquire preset first contour information;

[0047] The second determination module is specifically used for: performing image analysis on the target area to determine the contour information and the position of each icon in the target area; determining the target icon based on the first similarity between the contour information of each icon and the first contour information; wherein the first similarity between the contour information of the target icon and the first contour information is greater than a preset threshold; and determining the first position of the target icon based on the position of the target icon and each icon.

[0048] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, the second determination module is specifically used to: determine the target icon based on the first similarity and position information; wherein the position information includes the position of the target text and the position of each icon.

[0049] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, the second determination module is specifically used to: determine the first icon among the icons whose first similarity is greater than a preset threshold; determine the target distance between the target text and the first icon based on the position of the target text and the position of the first icon; if the target distance is less than the preset distance, and the first icon and the target text are in an aligned state in a first direction, determine the first icon as the target icon; wherein the first direction includes a horizontal direction or a vertical direction.

[0050] In combination with the second aspect and the above-mentioned implementation methods, in some implementation methods of the second aspect, the first determination module is specifically used to: perform optical character recognition on the image of the current display interface to determine the initial area where the target text is located; and expand the initial area based on the first pixel value to obtain the target area.

[0051] In combination with the second aspect and the above-mentioned implementation methods, in certain implementation methods of the second aspect, if the target control instruction indicates a click operation on the target interface element, the execution module is specifically used to: determine the center point of the target interface element; call the system's application program interface to send a click event to the center point; based on the click event, perform a click operation at the center point of the target interface element.

[0052] In combination with the second aspect and the above implementations, in some implementations of the second aspect, the execution module is further used to: if the target text does not exist in the current display interface and / or the target icon does not exist in the current display interface, execute a first operation to obtain a second display interface; wherein the first operation is used to switch the current display interface;

[0053] The third determination module is further used to determine a target position of a target interface element in the second display interface.

[0054] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0055] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0056] In a fifth aspect, a computer-readable storage medium is provided, which stores instructions. When the instructions are executed on an electronic device, the electronic device executes the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a schematic diagram of a display interface provided in an embodiment of the present application;

[0058] Figure 2 is a schematic flow chart of a method for controlling an interface element provided in an embodiment of the present application;

[0059] Figure 3 is a schematic flow chart of another method for controlling an interface element provided in an embodiment of the present application;

[0060] Figure 4 is a structural schematic diagram of a control device for an interface element provided in an embodiment of the present application;

[0061] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0063] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0064] In the prior art, fixed coordinates or preset identifiers are usually relied on to locate and operate interface elements in the display interface; however, the coordinates of interface elements may be different in different resolutions, different system versions or different devices, and the interface elements may be in dynamic change. If fixed coordinates or preset identifiers are used, the interface elements may be inaccurately located and the correct operations cannot be performed on the interface elements.

[0065] For example, in a display interface, there may be multiple target texts, such as Figure 1 "Bluetooth" and "Bluetooth List" in the , and there may be multiple icons with similar shapes, such as Figure 1 Icon 121, icon 123 and icon 124 in the display interface; if only the text is recognized, or the shape of the icon is recognized, it may lead to inaccurate recognition of the target interface element; therefore, how to accurately identify the target interface element in the display interface and determine the position of the target interface element is a technical problem that needs to be solved at present.

[0066] In view of this, the present application provides a method, device, electronic device and storage medium for controlling an interface element. Through the embodiments of the present application, when a target control instruction for a target interface element is detected, the position of a target icon corresponding to the target interface element is determined based on the image of the currently displayed interface, and a control operation is performed on the target interface element; thereby achieving accurate positioning and operation of the target interface element.

[0067] Figure 1 It is a schematic diagram of a display interface provided in an embodiment of the present application.

[0068] For example, Figure 1 The display interface 100 is a schematic diagram of the display interface of the terminal device, and the display interface 100 includes a screen display area 110, a target area 120, icons 121, 122, 123 and 124; wherein the target area 120 contains a target text.

[0069] For example, if a target control instruction for a target interface element on the screen is detected, the target interface element in the real interface is located and operated according to the target control instruction. Figure 1As shown, when the target control instruction is "turn on the Bluetooth switch", the target icon corresponding to the Bluetooth switch is determined by identifying the display interface, and the position of the target icon is determined as the position of the Bluetooth switch, that is, the position of icon 121 is determined; and a click operation is performed at icon 121 to turn on the Bluetooth switch.

[0070] For example, in the field of automated testing, the target control instruction may be a target test instruction for the current display interface. According to the target test instruction, the target interface elements in the display interface are located and operated to timely discover and correct various problems; Figure 1 As shown, when the target test instruction is "turn on the Bluetooth switch", the target icon corresponding to the Bluetooth switch (i.e., icon 121) is determined by identifying the display interface, and the position of icon 121 is determined as the position of the Bluetooth switch; a click operation is performed at icon 121 to execute the test operation corresponding to the target test instruction.

[0071] It should be noted that the above display interface 100 is an example of a display interface of a terminal device; this application does not limit this.

[0072] Combine the following Figure 2 and Figure 3 The flowchart in the figure describes in detail the control method of the interface elements provided in the embodiment of the present application.

[0073] Figure 2 It is a schematic flow chart of a method for controlling an interface element provided in an embodiment of the present application.

[0074] For example, Figure 2 The method 200 shown can be executed by a terminal device including a display screen; optionally, the control method of the interface element of the embodiment of the present application can also be applied to the vehicle system of the vehicle.

[0075] like Figure 2 As shown, the control method 200 of the interface element includes S210 to S240, and S210 to S240 are described in detail below.

[0076] S210: If a target control instruction for a target interface element in the current display interface is detected, an image of the current display interface is acquired.

[0077] Among them, the target control instruction is used to indicate the control operation of the target interface element in the current display interface; the control operation of the target interface element includes but is not limited to clicking operation, dragging operation (moving operation), etc. on the target interface element; the interface element is an element that can be output and displayed in the display interface of the terminal screen.

[0078] Exemplarily, when a target control instruction is detected, the system calls a screenshot application programming interface (Application Programming Interface, API) of the terminal device, and the screenshot API is used to obtain the image in the current screen; that is, the image of the current display interface.

[0079] Optionally, when a target control instruction is detected, the target control instruction is parsed to determine the target interface element corresponding to the target control instruction and the control operation on the target interface element; for example, when the target control instruction is "Please help me turn on the current Bluetooth switch"; by parsing the target control instruction, it is determined that the target interface element is the Bluetooth switch; the target control operation is a click operation on the target interface element (i.e., the Bluetooth switch).

[0080] S220, based on the image of the current display interface, determine a target area of ​​the target text corresponding to the target interface element in the current display interface.

[0081] Exemplarily, based on the image of the current display interface, the target area of ​​the target text in the current display interface is determined; wherein the target text is the text corresponding to the target interface element. The target text corresponding to the target interface element is determined by parsing the target control instruction; for example, if the target control instruction is "turn on the Bluetooth switch", the target text is "Bluetooth"; if the target control instruction is "turn on the mobile data switch", the target text is "mobile data".

[0082] Exemplarily, the target area is an area containing the target text; for example, Figure 1 As shown, when the target text is "Bluetooth", the target area is the area containing "Bluetooth", that is, the target area is the 120 area.

[0083] It should be noted that the target area may be an area including multiple target texts, such as Figure 1 As shown, the target area is 120 areas; it can also be divided according to each target text to determine multiple target areas; wherein each target area contains a target text; for example, according to Figure 1 Two target areas are determined, namely, a target area including "Bluetooth" and a target area including "Bluetooth list".

[0084] In one implementation, the above-mentioned determining the target area of ​​the text corresponding to the target interface element in the current display interface includes:

[0085] Optical character recognition is performed on the image of the current display interface to determine an initial area where the target text is located; and expansion processing is performed on the initial area based on the first pixel value to obtain a target area containing the target text.

[0086] Exemplarily, based on the image of the current display interface, optical character recognition (OCR) is performed on the image of the display interface to determine the initial area where the target text is located; the initial area is expanded to obtain a target area containing the target text. The OCR technology is used to accurately locate the target text, breaking through the limitations of traditional fixed coordinate positioning and improving the accuracy of locating the target interface elements.

[0087] Among them, OCR technology is an important text processing technology, which is usually used in document digitization, data extraction, identity authentication and other fields. Through image preprocessing, feature extraction, pattern matching and post-processing, OCR can recognize the text in the image and convert it into an editable text format.

[0088] Exemplarily, the image of the current display interface is analyzed through OCR technology, and the OCR engine can recognize all the text in the image and return the content and location coordinates of each text; the system screens and determines the target text and the location coordinates of the target text, and determines the initial area where the target text is located with the location coordinates of the target text as the center and according to the preset length and preset width; and expands the initial area to obtain the target area.

[0089] For example, the target control instruction is "turn on the Bluetooth switch", and the target text is "Bluetooth"; the position coordinates of the detected target text are expanded; if the coordinates of the initial "Bluetooth" text are (x, y, width, height), where x and y are the horizontal and vertical coordinates of the center point of the "Bluetooth" text, width represents the width of the initial area, and height represents the height of the initial area; the initial area is expanded 200 pixels in all directions, and the target area is (x, y, width+400, height+400). Use the new coordinates to capture a sub-image from the original screenshot (the image of the current display interface), that is, to capture the target area. The purpose of this step is to obtain the image area containing the "Bluetooth" text and its surrounding context to provide more information for subsequent semantic analysis.

[0090] It should be noted that the above is an example of target control instructions and text coordinates; this application does not impose any limitation on this.

[0091] In an embodiment of the present application, optical character recognition is performed on the image of the current display interface to determine the initial area where the target text is located; and the initial area is expanded to obtain a target area containing the target text; since the initial area is the area where the target text is located; the target area obtained by expanding the initial area includes the target text and the image area around the target text; that is, a target area containing more information is obtained, which facilitates image analysis of the target area and determines the target icon in the target area.

[0092] S230: Determine a first position of a target icon in the target area.

[0093] Among them, the target icon is the icon corresponding to the target interface element.

[0094] Exemplarily, image recognition is performed on the target area to determine the target icon in the target area; and the first position of the target icon is determined. The method for determining the target icon and the first position of the target icon is further described below.

[0095] In one implementation, a preset first outline information is obtained; wherein the first outline information is outline information corresponding to a preset target interface element;

[0096] Perform image analysis on the target area to determine the contour information and position of each icon in the target area; determine the target icon among the icons based on the first similarity between the contour information of each icon and the first contour information; wherein the first similarity between the contour information of the target icon and the first contour information is greater than a preset threshold; determine the first position of the target icon based on the position of the target icon and each icon.

[0097] Exemplarily, based on image analysis of the target area, the contour information of each icon in the target area and the position of each icon are determined; the first similarity between the contour information of each icon and the first contour information is determined; based on the first similarity, the target icon among the icons is determined; since the first contour information is the contour information corresponding to the preset target interface element; when the first similarity between the contour information of the icon and the first contour information is high, it indicates that the icon may be the target icon; therefore, the target icon among the icons is determined based on the first similarity; ensuring that the obtained target icon is the icon corresponding to the target interface element.

[0098] Exemplarily, the outline information of each icon is compared with the preset first outline information. When comparing, the comparison can be performed pixel by pixel or according to the pixel average. Optionally, a traditional digital image processing algorithm can be used to search for the icon corresponding to the first outline information in the target area, and the first position of the target icon can be determined according to the comparison result.

[0099] For example, when the target interface element is a Bluetooth switch, the preset first outline information is the outline information of the switch icon. Figure 1 The outline information shown in icons 121, 123 and 124 in FIG. 124; when the target interface element is a drop-down list, the preset first outline information is Figure 1 By identifying the target control instruction, the target interface element corresponding to the target control instruction and the outline information of the icon corresponding to the target interface element are determined.

[0100] It should be noted that the above is an example of the first contour information and the target interface element; in actual operation, the contour information of the interface element matching the target control instruction can be determined according to different target control instructions; this application does not limit this.

[0101] In a possible implementation manner, a target icon among the icons is determined based on a first similarity of the outline information and a second similarity of the color information.

[0102] Exemplarily, the preset first color information corresponding to the target interface element is obtained; the second similarity between the color information of each icon and the first color information is determined; based on the first similarity and the second similarity, the target icon among the icons is determined; wherein the first similarity between the contour information of the target icon and the first contour information is greater than a preset threshold, and the second similarity between the color information of the target icon and the first color information is greater than the first threshold. It is ensured that the target interface element in the current display interface can be more accurately identified based on the contour information and the color information.

[0103] Optionally, when performing image analysis on the target area, a multimodal large model technology can be used to analyze the target area; the captured image of the target area is input into the multimodal large model, which can process the image and text information at the same time to determine the target icon in the target area.

[0104] It should be noted that multimodal large model technology refers to large machine learning models that can process and integrate multiple different types of data. Traditional machine learning models usually focus on a single type of data, such as text, images, or audio. However, multimodal large models can process multiple types of data at the same time, such as text and images, text and audio, images and videos, etc., thereby achieving richer information representation and more accurate information extraction.

[0105] In one implementation, the step of determining the target icon based on the first similarity between the outline information of each icon and the first outline information includes:

[0106] Based on the first similarity and the position information, a target icon is determined; wherein the position information includes the position of the target text and the position of each icon.

[0107] Exemplarily, based on the first similarity and the position information, a target icon is determined among the icons; since there may be multiple icons with similar shapes in the target area; that is, there may be multiple icons with first similarities greater than a preset threshold in the target area; for example, Figure 1 As shown, when the target interface element is a Bluetooth switch, there are multiple switch icons of similar shapes in the display interface, including icon 121, icon 123 and icon 124; therefore, based on the first similarity and position information, the target icon is determined; ensuring that when there are multiple icons of similar shapes in the target area, the target icon corresponding to the target interface element can be determined.

[0108] In one implementation, the step of determining the target icon among the icons based on the first similarity and the position information includes:

[0109] Determine a first icon among the icons whose first similarity is greater than a preset threshold;

[0110] Determining a target distance between the target text and the first icon based on the position of the target text and the position of the first icon;

[0111] If the target distance is less than the preset distance, and the first icon and the target text are aligned in a first direction, the first icon is determined as the target icon; wherein the first direction includes a horizontal direction or a vertical direction.

[0112] Exemplarily, among each icon, a first icon having a first similarity greater than a preset threshold is determined; and a target distance between the target text and the first icon is determined; if the target distance is less than the preset distance, and the first icon and the target text are aligned in the horizontal direction or the vertical direction, the first icon is determined as the target icon; because in the display interface, the text corresponding to the interface element and the icon corresponding to the interface element are usually close to each other and are aligned in the horizontal direction or the vertical direction; therefore, if the target distance between the first icon and the target text is less than the preset distance, and they are aligned in the first direction (horizontally or vertically), it indicates that the first icon is the icon corresponding to the target text; that is, the first icon is the target icon.

[0113] For example, when the target interface element is a Bluetooth switch, based on the first similarity between the contour information of each icon and the first contour information, it is determined that the first icons with a first similarity greater than a preset threshold include icon 121, icon 123 and icon 124; if the target distance between icon 121 and the target text "Bluetooth" is 3 cm, the target distance between icon 123 and the target text is 5 cm, and the target distance between icon 124 and the target text is 7 cm; if the preset distance is 5 cm; then only the distance between icon 121 and the target text "Bluetooth" is less than the preset distance, and icon 121 and the target text are aligned in the horizontal direction; therefore, icon 121 is determined as the target icon; that is, icon 121 is the Bluetooth switch icon.

[0114] It should be noted that the above is an example of the distance between the target interface element and the target; this application does not impose any limitation on this.

[0115] S240, determining the first position as the target position of the target interface element; and executing a control operation corresponding to the target control instruction on the target interface element at the target position.

[0116] Exemplarily, the first position of the target icon is determined as the target position of the target interface element.

[0117] In one implementation, if the target control instruction indicates a click operation on the target interface element; performing the control operation corresponding to the target control instruction on the target interface element at the target position includes:

[0118] Determine the center point of the target interface element; call the system application program interface to send a click event to the center point; based on the click event, perform a click operation at the center point of the target interface element.

[0119] Exemplarily, the center point of the target interface element is determined, that is, the center point coordinates of the target icon are determined, and a system API is called to send a click event to the center point coordinates; the click event can trigger a click operation on the target element.

[0120] In an embodiment of the present application, if the target control instruction is a click operation on the target interface element, the center point of the target interface element is determined, the system's application program interface is called, and a click event is sent to the center point; and based on the click event, a click operation is performed at the center point of the target interface element; ensuring that the control operation corresponding to the target control instruction can be performed on the target interface element to change the state of the target interface element.

[0121] In one implementation, it also includes: if there is no target text in the current display interface, and / or there is no target icon in the current display interface, executing a first operation to obtain a second display interface; wherein the first operation is used to switch the current display interface; and determining a target position of the target interface element in the second display interface.

[0122] Exemplarily, if the target text does not exist in the current display interface, or the target icon does not exist in the current display interface, the first operation is performed to obtain the second display interface; the first operation is used to switch the current display interface; the first operation includes but is not limited to the sliding operation of the display interface, the return operation, and the operation of clicking to enter the lower-level page. In an embodiment of the present application, the second display interface is obtained by performing the first operation, and the target position of the target interface element is determined in the second display interface; when the target interface element does not exist in the current display interface, the display interface is switched, and the target position of the target interface element is determined in the second display interface after switching.

[0123] Through the above method steps, the target interface element (such as the Bluetooth switch) can be intelligently located and operated, and the user's target control instruction can be completed. This method combines image processing, text recognition, semantic understanding and precise coordinate operation to improve the accuracy of locating the target interface element.

[0124] In the above embodiment, an image of the current display interface is obtained, and a target area in the current display interface in which a target text corresponding to a target interface element is located is determined; and a first position of a target icon is determined in the target area; since there may be multiple icons of similar shapes in the display interface; for example, when the target control instruction is "turn on the Bluetooth switch", there may be multiple switch icons of similar shapes in the display interface; therefore, the target area in which the target text is located is determined, and the first position of the target icon is determined in the target area; this avoids the situation in which the target icon corresponding to the target interface element cannot be accurately identified when there are multiple icons of similar shapes in the current display interface; ensures accurate identification of the target icon in the current display interface; that is, accurately identifies the target interface element in the current display interface; determines the first position as the target position of the target interface element; and executes the corresponding control operation of the target control instruction on the target interface element at the target position; thereby being able to accurately locate the target interface element in the current display interface and execute the corresponding control operation.

[0125] Figure 3 It is a schematic flow chart of another method for controlling interface elements provided in an embodiment of the present application.

[0126] Figure 3The method 300 shown can be executed by a terminal device including a display screen; for example, the terminal device includes but is not limited to a mobile phone, a computer, and a smart tablet; optionally, the control method of the interface element of the embodiment of the present application can also be applied to the vehicle system.

[0127] like Figure 3 As shown, the control method 300 of the interface element includes S301 to S309, and S301 to S309 are described in detail below.

[0128] S301: If a target control instruction is detected, an image and first outline information of a current display interface are obtained.

[0129] Exemplarily, if a target control instruction is detected, the target control instruction is parsed to determine the target interface element indicated by the target control instruction and the control operation on the target interface element; an image of the current display interface and first contour information are obtained; wherein the image of the current display interface serves as the basis for subsequent analysis and operation; the first contour information is used to determine the target icon corresponding to the target interface element.

[0130] Optionally, the implementation of S301 can refer to Figure 2 The relevant description of S210 is not repeated here.

[0131] S302, performing optical character recognition on the image of the current display interface to determine an initial area of ​​the target text.

[0132] The target text is the text corresponding to the target interface element; the target interface element and the target text corresponding to the target interface element are determined by parsing the target control instruction.

[0133] Exemplarily, optical character recognition is performed on the image of the current display interface, and the image of the current display interface is analyzed by optical character recognition technology to identify all texts in the image. The target text is selected from all texts in the image; thereby determining the initial area where the target text is located, that is, determining the position coordinates of the target text, and determining the initial area where the target text is located with the position coordinates of the target text as the center and according to a preset length and a preset width.

[0134] S303: Expand the initial area to obtain a target area containing the target text.

[0135] Exemplarily, the initial region is expanded based on the first pixel value to obtain the target region. Since the initial region is a region divided according to a preset length and a preset width with the position coordinates of the target text as the center, the initial region has a small area and contains less information; therefore, the initial region is expanded to obtain the target region; that is, the image region containing the target text and its surrounding context is obtained to provide more information for subsequent image analysis.

[0136] S304: Perform image analysis on the target area to determine contour information and positions of each icon in the target area.

[0137] Exemplarily, image analysis is performed on the target area to determine the contour information and position of each icon in the target area. Since the target area may contain multiple icons, the contour information and position of each icon in the target area are determined by performing image analysis on the target area. This facilitates screening out the target icon corresponding to the target interface element from the icons.

[0138] S305: Determine a first similarity between the outline information of each icon and the first outline information.

[0139] Exemplarily, the first similarity between the contour information of each icon and the first contour information is determined, that is, the contour information of each icon is compared with the preset first contour information. When comparing, the comparison can be performed pixel by pixel or according to the pixel mean.

[0140] S306: Determine a target icon among the icons based on the first similarity.

[0141] Exemplarily, since the first contour information is the contour information corresponding to the preset target interface element; when the first similarity between the contour information of the icon and the first contour information is high, it indicates that the icon may be the target icon; therefore, the target icon among each icon is determined based on the first similarity; ensuring that the obtained target icon is the icon corresponding to the target interface element.

[0142] S307: Determine the first position of the target icon.

[0143] S308: Determine the first position as the target position of the target interface element.

[0144] Exemplarily, a target icon and a first position of the target icon are determined; since the target icon is an icon corresponding to the target interface element; therefore, the first position of the target icon is determined as the target position of the target interface element.

[0145] S309, executing a control operation corresponding to the target control instruction on the target interface element.

[0146] Exemplarily, the control operation corresponding to the target control instruction is performed on the target interface element at the target position; the control operation is the operation indicated by the target control instruction. The target control instruction is executed by performing the control operation on the target interface element.

[0147] Optionally, the implementation of S308 and S309 can refer to Figure 2 The relevant description of S240 is not repeated here.

[0148] In an embodiment of the present application, if a target control instruction is detected, an image of the current display interface is obtained; optical character recognition is performed on the image of the current display interface to determine the target area where the target text is located, and the first position of the target icon is determined in the target area; compared with the prior art of using fixed coordinates to locate interface elements, this solution is based on parsing the target control instruction and analyzing the image of the current display interface to identify the target icon corresponding to the target interface element; ensure accurate identification of the target icon in the current display interface; that is, accurately identify the target interface element in the current display interface.

[0149] Optionally, the multimodal large model technology can also be used to analyze the target area to determine the target icon in the target area; for example, if the target interface element is a Bluetooth switch, all images of the target area containing the target text "Bluetooth" are input into the multimodal large model; the model analyzes the image of each target area to determine which image's icon corresponding to the target text "Bluetooth" is most likely to be a switch control; factors that the model may consider include: 1. Whether there are visual elements similar to switch icons in the image of the target area; 2. The relative position of the target text "Bluetooth" and the visual element of the switch icon; 3. Other contextual information in the image (such as whether it is in the settings page).

[0150] Exemplarily, when determining whether there are visual elements similar to switch icons in the image of the target area, the multimodal model uses an edge detection algorithm to identify rectangular or circular outlines, and uses color analysis to identify typical switch state colors (for example, blue for on, gray for off), as well as shape template matching and other techniques; that is, searches for the icon corresponding to the first outline information in the target area, and determines the first position of the target icon based on the comparison results.

[0151] Exemplarily, when determining the relative position of the visual element of the "Bluetooth" text and the switch icon, the coordinates of the "Bluetooth" text are compared with the coordinates of the switch icon. The comparison algorithm may consider factors including: the distance between the target text "Bluetooth" and the switch icon (horizontal distance and vertical distance); the relative position relationship between the target text "Bluetooth" and the switch icon (for example, the target text is usually located on the left side of the switch icon); and the alignment of the target text "Bluetooth" and the switch icon (for example, the target text and the switch icon are usually aligned in the horizontal direction or vertical direction). Through this comparison, the switch icon of the Bluetooth switch is determined; that is, the target icon corresponding to the target interface element is determined.

[0152] It should be noted that the above is an example of the target interface element being a Bluetooth switch and the target text being "Bluetooth", and this application does not make any specific limitations on this. For other target interface elements and target texts; for example, in other scenarios where interface elements interact, such as Wi-Fi, location services, and other switch operations; all can be analyzed through a multimodal large model, and the same method can be used to locate the target interface element; no further details will be given here.

[0153] In the embodiments of the present application, a multimodal large model is used to perform semantic analysis on the image, and the interface element that is most likely to be a switch icon is intelligently identified, which greatly improves the accuracy of recognition; the text coordinates are compared with the icon coordinates to achieve more accurate positioning of the interface elements, effectively solving the problem of coexistence of multiple similar interface elements. When applied in the field of automated testing, it can improve the degree of automation of automated testing, reduce manual intervention, and thus improve test efficiency; that is, there is no need to pre-specify the coordinates of the target interface elements, and it can adapt to changes in different devices and system versions, greatly improving the stability and reliability of test cases.

[0154] Combination of the above Figures 1 to 3 The control method of the interface element provided by the embodiment of the present application is described in detail; Figure 4 and Figure 5 The device embodiments of the present application are described in detail. It should be understood that the device in the embodiments of the present application can execute the various methods of the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0155] Figure 4 It is a structural schematic diagram of a control device for an interface element provided in an embodiment of the present application.

[0156] For example, Figure 4 As shown, the control device 400 of the interface element includes:

[0157] The acquisition module 410 is used to acquire the image of the current display interface if a target control instruction for a target interface element in the current display interface is detected;

[0158] A first determination module 420 is used to determine a target area of ​​a target text corresponding to a target interface element in the current display interface based on the image of the current display interface;

[0159] The second determination module 430 is used to determine a first position of a target icon in the target area; wherein the target icon is an icon corresponding to the target interface element;

[0160] A third determining module 440, configured to determine the first position as a target position of a target interface element;

[0161] The execution module 450 is used to execute the control operation corresponding to the target control instruction on the target interface element at the target position.

[0162] Optionally, as an embodiment, the acquisition module 410 is further used to: acquire preset first contour information;

[0163] The second determination module 430 is specifically used for: performing image analysis on the target area to determine the contour information and the position of each icon in the target area; determining the target icon among the icons based on the first similarity between the contour information of each icon and the first contour information; wherein the first similarity between the contour information of the target icon and the first contour information is greater than a preset threshold; and determining the first position of the target icon based on the position of the target icon and each icon.

[0164] Optionally, as an embodiment, the second determination module 430 is specifically used to: determine the target icon based on the first similarity and the position information; wherein the position information includes the position of the target text and the position of each icon.

[0165] Optionally, as an embodiment, the second determination module 430 is specifically used to: determine the first icon among the icons whose first similarity is greater than a preset threshold; determine the target distance between the target text and the first icon based on the position of the target text and the position of the first icon; if the target distance is less than the preset distance, and the first icon and the target text are aligned in a first direction, determine the first icon as the target icon; wherein the first direction includes a horizontal direction or a vertical direction.

[0166] Optionally, as an embodiment, the first determination module 420 is specifically used to: perform optical character recognition on the image of the current display interface to determine an initial area where the target text is located; and expand the initial area based on the first pixel value to obtain the target area.

[0167] Optionally, as an embodiment, if the target control instruction indicates a click operation on the target interface element, the execution module 450 is specifically used to: determine the center point of the target interface element; call the system's application program interface to send a click event to the center point; and based on the click event, perform a click operation at the center point of the target interface element.

[0168] Optionally, as an embodiment, the execution module 450 is further used to: if the target text does not exist in the current display interface, and / or the target icon does not exist in the current display interface, execute the first operation to obtain the second display interface; wherein the first operation is used to switch the current display interface;

[0169] The third determination module 440 is further used to determine a target position of a target interface element in the second display interface.

[0170] It should be noted that the control device of the above interface elements is embodied in the form of functional units. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0171] For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0172] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0173] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0174] Exemplarily, the electronic device 500 includes: a processor 510 , a memory 520 , and an executable program code 530 .

[0175] Exemplarily, the electronic device 500 includes one or more processors 510, which can support the electronic device 500 to implement the control method of the interface element in the method embodiment. The processor 510 can be a general-purpose processor or a special-purpose processor. For example, the processor 510 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0176] Exemplarily, the processor 510 may be used to control the electronic device 500, execute software programs, and process data of the software programs. The electronic device 500 may also include a communication unit to implement input (reception) and output (transmission) of signals.

[0177] Exemplarily, the electronic device 500 may include one or more memories 520 on which executable program code 530 is stored. The executable program code 530 can be executed by the processor 510 to generate instructions so that the processor 510 executes the control method of the interface elements described in the above method embodiment according to the instructions.

[0178] Optionally, data may be stored in the memory 520. Optionally, the processor 510 may read data stored in the memory 520, which may be stored at the same storage address as the executable program code 530, or may be stored at a different storage address from the executable program code 530.

[0179] Exemplarily, the processor 510 and the memory 520 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0180] Exemplarily, the memory 520 can be used to store relevant programs of the method for controlling interface elements provided in the embodiments of the present application, and the processor 520 can be used to call the executable program code 530 stored in the memory 520 when controlling the electronic device to execute the method for controlling interface elements in the embodiments of the present application; for example, if a target control instruction for a target interface element in the current display interface is detected, an image of the current display interface is acquired; based on the image of the current display interface, a target area in the current display interface corresponding to the target text of the target interface element is determined; in the target area, a first position of the target icon is determined; wherein the target icon is an icon corresponding to the target interface element; the first position is determined as the target position of the target interface element; and a control operation corresponding to the target control instruction is performed on the target interface element at the target position.

[0181] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for controlling the interface elements of any of the aforementioned embodiments.

[0182] Among them, computer-readable storage media may include but are not limited to any type of disk, including floppy disks, optical disks, digital versatile disks (Digital Video Disc, DVD), compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), microdrives and magneto-optical disks, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable programmable read only memory (Electrically Erasable Programmable read only memory, EEPROM), dynamic random access memory (Dynamic Random Access Memory, DRAM), video random access memory (Video Random Access Memory, VRAM), flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0183] The present application also provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above-mentioned related steps to implement a method for controlling an interface element in the above-mentioned embodiment.

[0184] In addition, the electronic device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the electronic device may include a connected processor and a memory; wherein the memory is used to store instructions, and when the electronic device is running, the processor may call and execute instructions so that the chip executes a control method of an interface element in the above-mentioned embodiments.

[0185] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this application are all used to execute the control method of the corresponding interface elements provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the control method of the corresponding interface elements provided above, and will not be repeated here.

[0186] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0187] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for controlling an interface element, characterized in that: The method comprises: If a target control instruction for a target interface element in the current display interface is detected, an image of the current display interface is acquired; Based on the image of the current display interface, determining a target area of ​​the target text corresponding to the target interface element in the current display interface; In the target area, determining a first position of a target icon; wherein the target icon is an icon corresponding to the target interface element; The first position is determined as the target position of the target interface element; and a control operation corresponding to the target control instruction is performed on the target interface element at the target position.

2. The method according to claim 1, characterized in that Also includes: Obtaining preset first contour information; Determining a first position of the target icon in the target area includes: Performing image analysis on the target area to determine contour information of each icon in the target area and the position of each icon; Determine the target icon based on the first similarity between the outline information of each icon and the first outline information; wherein the first similarity between the outline information of the target icon and the first outline information is greater than a preset threshold; Based on the positions of the target icon and the icons, a first position of the target icon is determined.

3. The method according to claim 2, characterized in that The determining the target icon based on the first similarity between the outline information of each icon and the first outline information includes: The target icon is determined based on the first similarity and position information; wherein the position information includes the position of the target text and the positions of the icons.

4. The method according to claim 3, characterized in that: The determining the target icon based on the first similarity and the position information includes: Determine a first icon among the icons whose first similarity is greater than the preset threshold; Determining a target distance between the target text and the first icon based on a position of the target text and a position of the first icon; If the target distance is less than a preset distance, and the first icon and the target text are aligned in a first direction, the first icon is determined as the target icon; wherein the first direction includes a horizontal direction or a vertical direction.

5. The method according to claim 1, characterized in that The determining of a target area of ​​text corresponding to the target interface element in the current display interface includes: Performing optical character recognition on the image of the current display interface to determine an initial area where the target text is located; The initial area is expanded based on the first pixel value to obtain the target area.

6. The method according to any one of claims 1 to 5, characterized in that If the target control instruction indicates a click operation on a target interface element; performing a control operation corresponding to the target control instruction on the target interface element at the target position includes: Determine the center point of the target interface element; Calling the system's application program interface to send a click event to the central point; Based on the click event, the click operation is performed at the center point of the target interface element.

7. The method according to claim 1, characterized in that Also includes: If the target text does not exist in the current display interface, and / or the target icon does not exist in the current display interface, performing a first operation to obtain a second display interface; wherein the first operation is used to switch the current display interface; Determine a target position of the target interface element in the second display interface.

8. A control device for an interface element, characterized in that: The device comprises: An acquisition module, configured to acquire an image of the current display interface if a target control instruction for a target interface element in the current display interface is detected; A first determination module, configured to determine a target area of ​​the target text corresponding to the target interface element in the current display interface based on the image of the current display interface; A second determination module is used to determine a first position of a target icon in the target area; wherein the target icon is an icon corresponding to the target interface element; A third determining module, used to determine the first position as the target position of the target interface element; An execution module is used to execute the control operation corresponding to the target control instruction on the target interface element at the target position.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 7.

Images