Control positioning method, device, electronic device and readable storage medium

Abstract

The present application discloses a control positioning method, device, electronic device and readable storage medium, the method comprising: receiving first description information of a target control of a pre-stored target control instruction; respectively obtaining second description information of m controls to be positioned in an interface, the second description information comprising second feature information and second relationship information between the control to be positioned and a second reference control within a preset range thereof, the second feature information being used to indicate attribute information of the control to be positioned, the second relationship information being used to indicate a positional relationship between the control to be positioned and the second reference control, and m being an integer greater than 0; respectively obtaining similarity values ​​between the first description information and the second description information of the m controls to be positioned; determining that the control to be positioned corresponding to the target similarity value is the target control to be manipulated, the target similarity value being greater than a preset threshold, and the target similarity value being the maximum similarity value among the m similarity values ​​obtained.

Description

Technical Field

[0001] The present application belongs to the field of automated testing technology, and specifically relates to a control positioning method, device, electronic device and readable storage medium. Background Art

[0002] The key link in automated testing is control positioning, that is, according to the key information recorded in the script step, locate the target control to be operated in the step on the test device, and then perform the corresponding operation. The following are common control positioning methods.

[0003] Method 1: Record key information of the control, such as control type, identity document (ID), text or Extensible Markup Language Path Language (Xml Path Language, xpath), etc. When executing the script, find the location of the target control in the control tree through these key information;

[0004] Method 2: Use optical character recognition (OCR) technology or image recognition technology to find the target text or target image in the interface, thereby locating the target control.

[0005] Method three, using the surrounding information of the target control to calculate an offset, obtain a coordinate within the range of the target control, and thus locate the target control.

[0006] However, in the case of different key information, for example, different devices may have different xpaths. In this case, it is necessary to manually formulate rules to solve the impact caused by different xpaths. For solution 2, when the control has no obvious features, it is also necessary to manually formulate rules to locate the control to be controlled. As for solution 3, the applicable scenarios are also limited, and the offset also needs to be adjusted manually.

[0007] It can be seen that the above method for locating the control to be controlled has limited applicable scenarios and requires a lot of maintenance work. Summary of the invention

[0008] The purpose of the embodiments of the present application is to provide a control positioning method, device, electronic device and readable storage medium, which can solve the problem that the method of positioning the control to be controlled in the related art has limited applicable scenarios and requires a lot of maintenance work.

[0009] In a first aspect, an embodiment of the present application provides a control positioning method, comprising receiving first description information of a pre-stored target control, wherein the first description information comprises first characteristic information and first relationship information between the target control and a first reference control within a preset range thereof, the first characteristic information being used to indicate attribute information of the target control, and the first relationship information being used to indicate a positional relationship between the target control and the first reference control; obtaining second description information of m controls to be positioned in an interface respectively, wherein the second description information comprises second characteristic information and second relationship information between the control to be positioned and a second reference control within a preset range thereof, the second characteristic information being used to indicate attribute information of the control to be positioned, the second relationship information being used to indicate a positional relationship between the control to be positioned and the second reference control, and m being an integer greater than 0; obtaining similarity values ​​between the first description information and the second description information of the m controls to be positioned respectively; determining that the control to be positioned corresponding to the target similarity value is the target control, wherein the target similarity value is greater than a preset threshold value, and the target similarity value is the maximum similarity value among the m similarity values ​​obtained.

[0010] In a second aspect, an embodiment of the present application provides a control positioning device, including a receiving module, used to receive pre-stored first description information of a target control, wherein the first description information includes first feature information and first relationship information between the target control and a first reference control within a preset range thereof, the first feature information is used to indicate the attribute information of the target control, and the first relationship information is used to indicate the positional relationship between the target control and the first reference control; a first acquisition module, used to respectively acquire second description information of m controls to be positioned in an interface, wherein the second description information includes second feature information and second relationship information between the control to be positioned and a second reference control within a preset range thereof, the second feature information is used to indicate the attribute information of the control to be positioned, the second relationship information is used to indicate the positional relationship between the control to be positioned and the second reference control, and m is an integer greater than 0; a second acquisition module, used to respectively acquire similarity values ​​between the first description information and the second description information of the m controls to be positioned; a determination module, used to determine that the control to be positioned corresponding to the target similarity value is the target control, wherein the target similarity value is greater than a preset threshold value, and the target similarity value is the maximum similarity value among the acquired m similarity values.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0014] In the embodiment of the present application, by receiving the first description information of the pre-stored target control, and then obtaining the second description information of the control to be located in the terminal device interface, the first description information and the second description information of each control to be located in the interface are calculated to obtain the similarity value between the target control and each control to be located in the interface, and the control to be located corresponding to the maximum similarity value is determined as the target control, thereby accurately locating the target control. Since the feature information and relationship information included in the description information can be adapted to various scenarios, a large amount of manual maintenance is not required, thereby solving the problem that the method for locating the control to be controlled in the related art is limited in applicable scenarios and requires a large amount of maintenance work. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a control positioning method provided in an embodiment of the present application;

[0016] Figure 2 is a structural schematic diagram of a matching model provided in an embodiment of the present application;

[0017] Figure 3a is a schematic diagram of interface controls before the controls in the interface are moved in an embodiment of the present application;

[0018] Figure 3b is a schematic diagram of interface controls after controls in the interface are moved in an embodiment of the present application;

[0019] Figure 4a is a schematic diagram of the interface controls before the interface resolution is reduced in the embodiment of the present application;

[0020] Figure 4b is a schematic diagram of the interface controls after reducing the interface resolution in the embodiment of the present application;

[0021] Figure 5a is a schematic diagram of interface controls before increasing the interface resolution in an embodiment of the present application;

[0022] Figure 5b is a schematic diagram of the interface controls after the interface resolution is increased in the embodiment of the present application;

[0023] Figure 6ais a schematic diagram of the interface controls before the interface controls are missing in the embodiment of the present application;

[0024] Figure 6b is a schematic diagram of the interface controls before the interface controls are missing in the embodiment of the present application;

[0025] Figure 7 It is a structural schematic diagram of a control positioning device provided in an embodiment of the present application;

[0026] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The control positioning method, device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] Figure 1 1 is a flow chart of a control positioning method provided in an embodiment of the present application. The method 100 can be executed by a terminal device. In other words, the method can be executed by software or hardware installed in the terminal device. Figure 1 As shown, the method may include the following steps.

[0031] S110: Pre-store first description information of the target control.

[0032] Among them, the first description information includes first characteristic information and first relationship information between the target control and a first reference control within a preset range thereof, the first characteristic information is used to indicate the attribute information of the target control, and the first relationship information is used to indicate the positional relationship between the target control and the first reference control.

[0033] Specifically, in the automated test, the target control is recorded in the step of running the script, and the first description information of the target control is calculated in advance and stored in the terminal device.

[0034] S120: Obtain second description information of m controls to be located in the interface respectively.

[0035] Among them, the second description information includes second characteristic information and second relationship information between the control to be located and a second reference control within a preset range thereof, the second characteristic information is used to indicate the attribute information of the control to be located, the second relationship information is used to indicate the positional relationship between the control to be located and the reference control, and m is an integer greater than 0.

[0036] Specifically, the second feature information is used to indicate the attribute information of the control to be located, and the attribute information includes at least one of type information, size information, position information and image information. Type information refers to the type of the control to be located, for example, whether the control to be located is a button or a text box. Size information refers to the size of the control to be located. Position information refers to the coordinate position in the control to be located. For image information, the size of the control to be located can be reduced or enlarged to a preset value. For example, the size of the control to be located can be reduced to a small 8×8 picture. Since each pixel has three color values ​​of red, green and blue, 8×8×3 features are obtained.

[0037] The relationship information is used to indicate the positional relationship between the control to be located and the second reference control. The second reference control includes the left neighbor (the control to be located on the left), the right neighbor (the control to be located on the right), the upper neighbor (the control to be located above), and the lower neighbor (the control to be located below) of the control to be located. The positional relationship may include the distance between the control to be located and the second reference control. Specifically, taking the control to be controlled and one of the second reference controls as a box as an example, the distance between the two sides adjacent to the control to be controlled and the second reference control may be calculated, and the calculated distance may be used as the distance between the control to be located and the second reference control. The positional relationship may also include whether there is a second reference control within the preset range of the control to be located, and may also include the orientation of the control to be located in the second reference control.

[0038] The description information of the m controls to be located may be determined through the second characteristic information and the second relationship information of the controls to be located.

[0039] S130: Obtain similarity values ​​between the first description information and the second description information of the m controls to be located respectively.

[0040] It should be noted that before locating the control, the second description information of each control to be located in the interface can be calculated and stored. In this step, the first description information is compared one by one with the second description information of the m controls to be located calculated in step 120, and m similarity values ​​can be obtained.

[0041] S140: Determine that the to-be-located control corresponding to the target similarity value is the target control.

[0042] Among them, the target similarity value is greater than a preset threshold, and the target similarity value is the maximum similarity value among the m similarity values ​​obtained. If the target similarity value is not greater than the preset threshold, it means that none of the m controls to be located matches the control to be controlled.

[0043] The control positioning method provided in the embodiment of the present application receives the first description information of the pre-stored target control, obtains the second description information of the control to be positioned in the terminal device interface, calculates the first description information and the second description information of each control to be positioned in the interface, obtains the similarity value between the target control and each control to be positioned in the interface, and determines the control to be positioned corresponding to the maximum similarity value as the target control, thereby accurately positioning the target control. Since the feature information and relationship information included in the description information can be adapted to various scenarios, a large amount of manual maintenance is not required, thereby solving the problem that the method for positioning the control to be manipulated in the related art is limited in applicable scenarios and requires a large amount of maintenance work.

[0044] In a possible implementation, S120 may include: respectively acquiring second characteristic information of m controls to be located in the interface; for each control to be located, acquiring second relationship information between the control to be located and a second reference control within a preset range; and acquiring second description information of the control to be located based on the second characteristic information and the second relationship information. By acquiring the second characteristic information and the second relationship information, the second description information can be determined.

[0045] In a possible implementation, obtaining the second description information of the control to be located according to the second feature information and the second relationship information may include: performing vector conversion processing on the second feature information and the second relationship information, and using the obtained vector as the second description information of the control to be located.

[0046] In this possible implementation, after the feature information and relationship information are converted into vectors, the vector is obtained as the description information of the control to be located, and the similarity value between the control to be located and the control to be operated is calculated through the vector. For example, the obtained vector can be input into the matching model mentioned later to obtain the similarity value, so that the similarity value between the first description information and the second description information of the control to be located can be quickly obtained, thereby improving the efficiency of locating the target control.

[0047] In a possible implementation, respectively obtaining similarity values ​​between the first description information and the second description information of m controls to be located may include: inputting the first description information and the second description information of one control to be located into a matching model, and obtaining a similarity value between the target control and the control to be located by calculation of the matching model; inputting the first description information and the second description information of the next control to be located into the matching model, and obtaining a similarity value between the target control and the next control to be located by calculation of the matching model, and repeating this process to respectively obtain similarity values ​​between the first description information of the target control and the second description information of m controls to be located.

[0048] Figure 2 A structural diagram of the matching model is shown in Figure 2. Figure 2 As shown, the matching model has two input terminals. The second description information of the control to be located can be input as description information 1, and the first description information of the target control can be input as description information 2. The two description information are respectively extracted through neural network features to obtain higher-level features, and then connected together. After passing through several layers of neural networks, the final result is a scalar, which represents the similarity between the control to be located and the control to be manipulated.

[0049] The embodiment of the present application uses a matching model to calculate the similarity value between the first description information of the target control and the second description information of m controls to be located. Since the matching model is trained based on a large amount of data, there is no need to manually maintain a large number of complex rules. For script writers, there is no need to configure various complex parameters for script steps. Therefore, the difficulty and cost of maintaining the automation system can be reduced, while improving the efficiency of script writing and enhancing the stability of script execution.

[0050] Of course, in order to make the matching model converge faster and the training more stable, the second feature information and the second relationship information may be normalized first. Therefore, in a possible implementation, before the feature information and the relationship information are vectorized, the method may include: normalizing the second feature information and the second relationship information.

[0051] Specifically, for the text features in the second feature information, such as class, id, or text, you can first define a character set. This character set contains characters that appear at a certain frequency in the collected control tree. With the character set, you can normalize the text features. The specific method is: take a certain number of characters at the beginning and a certain number of characters at the end of the text. This number can be set to different values ​​for different features, get the sequence number of each character in the character set, and then divide it by the size of the character set to get a decimal between 0 and 1. For characters that are not in the character set and cannot be calculated, use a value of 0 instead. For insufficient characters, use 0 to fill in to ensure that the length of each feature is fixed.

[0052] For the coordinate data in the second feature information and the second relationship information, for example, the location information in the second feature information can be represented by the coordinates (x, y), and the position of the control to be located in the second relationship information relative to the second reference control can also be represented by the coordinates (x, y). For normalization of such coordinate data, the width of the interface can be divided by x, and the height of the screen can be divided by y to obtain normalized data.

[0053] For the information indicating yes or no in the second feature information and the second relationship information, for example, whether there is a control on the right of the control to be located, for this type of information, 1 can be used to indicate yes and 0 can be used to indicate no, thereby obtaining normalized data.

[0054] For the distance between the control to be located and the second reference control in the second relationship information, the distance may be divided by the diagonal distance of the interface to obtain normalized data.

[0055] For image information, the image matrix may be multiplied by a preset value, such as 1 / 255, to obtain normalized data.

[0056] It should be noted that, in addition to the above-mentioned method for normalizing various types of information, other methods may be used to normalize various types of information, and this application does not make any specific limitation on this.

[0057] After the second feature information and the second relationship information are normalized, the second feature information and the second relationship information are vector-converted, so that the similarity value between the target control and the control to be located can be calculated more efficiently, thereby accurately locating the target control.

[0058] Since the matching model is obtained by training a large amount of data, in one possible implementation, before receiving the first description information of the pre-stored target control, the method may include: obtaining similar information pairs and dissimilar information pairs; and training the matching model based on the similar information pairs and the dissimilar information pairs.

[0059] Furthermore, obtaining similar information pairs and dissimilar information pairs may include the following steps.

[0060] Step 1, obtain the third characteristic information of the current control.

[0061] The third characteristic information is used to indicate the attribute information of the current control.

[0062] Step 2: Obtain third relationship information between the current control and a third reference control within a preset range thereof.

[0063] The third relationship information is used to indicate the positional relationship between the current control and the third reference control.

[0064] Step 3: Perform vector conversion processing on the third characteristic information and the third relationship information, and use the obtained vector as the third description information of the current control.

[0065] Specifically, the third characteristic information and the third relationship information may be normalized first, and then the third characteristic information and the third relationship information may be transformed into vectors, and the obtained vectors may be used as the third description information of the current control.

[0066] Step 4: Use the method of steps 1 to 3 to obtain the third description information of other controls.

[0067] Step 5: Change the third characteristic information of the on-screen control and / or change the third relationship information of the on-screen control to obtain fourth characteristic information and fourth relationship information.

[0068] For example, you can move the controls in the interface to change the position of the current control, such as Figure 3a and Figure 3b As shown, Figure 3a This is the interface before moving. Figure 3b This is the interface after moving.

[0069] You can also change the resolution of the interface, such as Figure 4a and Figure 4b As shown, Figure 4a This is the interface before the resolution is reduced. Figure 4b This is the interface after the resolution is reduced. Figure 5a and Figure 5b As shown, Figure 5a This is the interface before the resolution is increased. Figure 5b This is the interface after the resolution is increased.

[0070] You can also simulate the situation where the control is missing, such as Figure 6a and Figure 6b As shown, Figure 6a This is the case where the controls in the interface are not missing. Figure 6b This is the case where controls are missing from the interface.

[0071] By changing the third characteristic information of the on-screen control and / or changing the third relationship information of the on-screen control, the change of the current control in various scenarios is simulated to obtain the fourth characteristic information and the fourth relationship information.

[0072] Step 6: Perform vector conversion processing on the fourth characteristic information and the fourth relationship information, and use the obtained vector as the fourth description information of the current control.

[0073] Similarly, the fourth characteristic information and the fourth relationship information may be normalized first, and then the fourth characteristic information and the fourth relationship information may be vector-converted to obtain the fourth description information of the current control.

[0074] Step 7: taking the third description information of the current control and the fourth description information of the current control as a similar information pair.

[0075] In a specific application, the similar information pair may be labeled as 1.0, indicating that the two description information are completely consistent.

[0076] Step 8: The third description information of the current control and the third description information of other controls are regarded as non-similar information pairs.

[0077] In a specific application, the non-similar information pair may be labeled as 0.0, indicating that the two description information are completely inconsistent.

[0078] Through the above steps, a pair of similar information and a pair of dissimilar information can be obtained, and the similar information pair and the dissimilar information pair can be input into the matching model for training.

[0079] Similarly, by using the above method, a large number of similar information pairs and dissimilar information pairs can be obtained, and the matching model can be trained using any deep learning framework. In specific applications, more scenarios can be simulated by changing the feature information and relationship information of each control, so that the matching model has a better effect.

[0080] It should be noted that the control positioning method provided in the embodiment of the present application can be executed by a control positioning device, or a control module in the control positioning device for executing the control positioning method. In the embodiment of the present application, the control positioning device provided in the embodiment of the present application is described by taking the control positioning method executed by the control positioning device as an example.

[0081] Figure 7 Schematic diagram of a control positioning device provided in an embodiment of the present application. Figure 7 As shown, the control positioning device 700 includes: a receiving module 710 , a first obtaining module 720 , a second obtaining module 730 and a determining module 740 .

[0082] A receiving module 710 is used to receive pre-stored first description information of a target control, wherein the first description information includes first feature information and first relationship information between the target control and a first reference control within a preset range thereof, the first feature information is used to indicate attribute information of the target control, and the first relationship information is used to indicate the positional relationship between the target control and the first reference control; a first acquisition module 720 is used to respectively acquire second description information of m controls to be located in the interface, wherein the second description information includes second feature information and second relationship information between the control to be located and a second reference control within a preset range thereof, the second feature information is used to indicate attribute information of the control to be located, the second relationship information is used to indicate the positional relationship between the control to be located and the second reference control, and m is an integer greater than 0; a second acquisition module 730 is used to respectively acquire similarity values ​​between the first description information and the second description information of the m controls to be located; a determination module 740 is used to determine that the control to be located corresponding to the target similarity value is the target control, wherein the target similarity value is greater than a preset threshold value, and the target similarity value is the maximum similarity value among the acquired m similarity values.

[0083] In one implementation, the first acquisition module 720 respectively acquires the second description information of m controls to be located in the interface, including: respectively acquiring the second characteristic information of m controls to be located in the interface; for each of the controls to be located, acquiring the second relationship information of the control to be located and a second reference control within a preset range thereof; and acquiring the second description information of the control to be located based on the second characteristic information and the second relationship information.

[0084] In one implementation, the first acquisition module 720 acquires second description information of the control to be located based on the second feature information and the second relationship information: performs vector conversion processing on the second feature information and the second relationship information, and uses the obtained vector as the description information of the control to be located.

[0085] In one implementation, the second acquisition module 730 respectively acquires similarity values ​​between the first description information and the second description information of m controls to be located, including: inputting the first description information and the second description information of one control to be located into a matching model, and obtaining a similarity value between the target control and the control to be located through calculation by the matching model; inputting the first description information and the second description information of the next control to be located into the matching model, and obtaining a similarity value between the target control and the next control to be located through calculation by the matching model, and repeating this process to respectively obtain similarity values ​​between the first description information of the target control and the second description information of m controls to be located.

[0086] In one implementation, the control locating device 700 may further include a processing module, and the processing module is used to normalize the second feature information and the second relationship information before performing vector conversion processing on the second feature information and the second relationship information.

[0087] In one implementation, the control locating device 700 may further include a training module, which is used to obtain similar information pairs and dissimilar information pairs before receiving the pre-stored first description information of the target control; and train the matching model based on the similar information pairs and the dissimilar information pairs.

[0088] In one implementation, the training module obtains similar information pairs and dissimilar information pairs including: step 1, obtaining the third feature information of the current control; step 2, obtaining the third relationship information between the current control and the third reference control within a preset range; step 3, performing vector conversion processing on the third feature information and the third relationship information, and using the obtained vector as the third description information of the current control; step 4, adopting the method of step 1-step 3 to obtain the third description information of other controls; step 5, changing the third feature information of the on-screen control and / or changing the third relationship information of the on-screen control to obtain fourth feature information and fourth relationship information; step 6, performing vector conversion processing on the fourth feature information and the fourth relationship information, and using the obtained vector as the fourth description information of the current control; step 7, using the third description information of the current control and the fourth description information of the current control as a similar information pair; step 8, using the third description information of the current control and the third description information of other controls as a dissimilar information pair.

[0089] The control positioning device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0090] The control positioning device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0091] The control positioning device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the method embodiment of FIG. 6 are not described again here.

[0092] Alternatively, if Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned control positioning method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0093] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0094] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned control positioning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0095] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0096] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned control positioning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0097] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0098] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned control positioning method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0099] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited, and the functions are performed in the order shown or discussed, and may also include performing the functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0101] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A control positioning method, characterized in that: include: Receive first description information of a pre-stored target control, wherein the first description information includes first characteristic information and first relationship information between the target control and a first reference control within a preset range thereof, the first characteristic information is used to indicate attribute information of the target control, and the first relationship information is used to indicate a positional relationship between the target control and the first reference control; Respectively obtain second description information of m controls to be located in the interface, wherein the second description information includes second characteristic information and second relationship information between the control to be located and a second reference control within a preset range thereof, the second characteristic information is used to indicate attribute information of the control to be located, the second relationship information is used to indicate a positional relationship between the control to be located and the second reference control, and m is an integer greater than 0; Respectively obtain similarity values ​​between the first description information and the second description information of the m controls to be located; Determine that a control to be located corresponding to a target similarity value is the target control, wherein the target similarity value is greater than a preset threshold, and the target similarity value is a maximum similarity value among the m similarity values ​​obtained.

2. The control positioning method according to claim 1, characterized in that: The second description information of m controls to be located in the interface is obtained respectively, including: Obtain the second characteristic information of m controls to be located in the interface respectively; For each of the controls to be located, obtaining second relationship information between the control to be located and a second reference control within a preset range thereof; According to the second characteristic information and the second relationship information, second description information of the control to be located is obtained.

3. The control positioning method according to claim 2, characterized in that: Acquiring second description information of the control to be located according to the second feature information and the second relationship information includes: The second feature information and the second relationship information are converted into vectors, and the obtained vector is used as description information of the control to be located.

4. The control positioning method according to claim 3, characterized in that: Respectively obtaining similarity values ​​between the first description information and the second description information of the m controls to be located, including: Inputting the first description information and the second description information of one of the to-be-located controls into a matching model, and obtaining a similarity value between the target control and the one of the to-be-located controls through calculation by the matching model; The first description information and the second description information of the next control to be located are input into the matching model, and the similarity value between the target control and the next control to be located is obtained through calculation of the matching model. This process is repeated to obtain the similarity values ​​between the first description information of the target control and the second description information of m controls to be located.

5. The control positioning method according to claim 4, characterized in that: Before performing vector conversion processing on the second feature information and the second relationship information, the method includes: The second feature information and the second relationship information are normalized.

6. The control positioning method according to claim 4, characterized in that: Before receiving the pre-stored first description information of the target control, the method includes: Obtain similar information pairs and dissimilar information pairs; The matching model is trained according to the similar information pairs and the non-similar information pairs.

7. The control positioning method according to claim 6, characterized in that: Get similar information pairs and dissimilar information pairs, including: Step 1, obtaining the third characteristic information of the current control; Step 2, obtaining third relationship information between the current control and a third reference control within a preset range thereof; Step 3, performing vector conversion processing on the third characteristic information and the third relationship information, and using the obtained vector as the third description information of the current control; Step 4, using the method of step 1 to step 3, obtain the third description information of other controls; Step 5: Change the third characteristic information of the current control and / or change the third relationship information of the current control to obtain fourth characteristic information and fourth relationship information; Step 6, performing vector conversion processing on the fourth characteristic information and the fourth relationship information, and using the obtained vector as the fourth description information of the current control; Step 7, taking the third description information of the current control and the fourth description information of the current control as a similar information pair; Step 8: The third description information of the current control and the third description information of other controls are regarded as non-similar information pairs.

8. A control positioning device, characterized in that: include: A receiving module, configured to receive pre-stored first description information of a target control, wherein the first description information includes first characteristic information and first relationship information between the target control and a first reference control within a preset range thereof, the first characteristic information being used to indicate attribute information of the target control, and the first relationship information being used to indicate a positional relationship between the target control and the first reference control; A first acquisition module is used to respectively acquire second description information of m controls to be located in the interface, wherein the second description information includes second feature information and second relationship information between the control to be located and a second reference control within a preset range thereof, the second feature information is used to indicate attribute information of the control to be located, the second relationship information is used to indicate a positional relationship between the control to be located and the second reference control, and m is an integer greater than 0; A second acquisition module, used to respectively acquire similarity values ​​between the first description information and the second description information of the m controls to be located; A determination module is used to determine that a control to be located corresponding to a target similarity value is the target control, wherein the target similarity value is greater than a preset threshold, and the target similarity value is a maximum similarity value among the m similarity values ​​obtained.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the control positioning method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the control positioning method according to any one of claims 1 to 7 are implemented.

Images